Cisco APIC NX-OS Style Command-Line Interface Configuration Guide First Published: 2015-12-08 Last Modified: 2017-08-31 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental. Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http:// www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R) Please send general FSF & GNU inquiries to [email protected] There are also other ways to contact the FSF. Please send broken links and other corrections or suggestions to [email protected] Please see the Translations README for information on coordinating and submitting translations of this article. Copyright © 2007, 2009, 2011 Free Software Foundation, Inc. Verbatim copying and distribution of this entire article are permitted worldwide, without royalty, in any medium, provided this notice, and the copyright notice, are preserved. Updated: Date: 2011/06/28 02:44:32 © 2015-2017 Cisco Systems, Inc. All rights reserved. CONTENTS Preface Preface xv Audience xv New and Changed Information xv Document Conventions xxiii Related Documentation xxiv Documentation Feedback xxv CHAPTER 1 Using the APIC CLI 1 Accessing the NX-OS Style CLI 1 Using the NX-OS Style CLI for APIC 2 Differences in Usage from NX-OS 4 Mixing the NX-OS Style CLI and Basic and Advanced APIC GUI 5 CHAPTER 2 Configuring Fabric and Interfaces 7 Fabric and Interface Configuration 7 Graceful Insertion and Removal (GIR) Mode 8 Removing a Switch to Maintenance Mode Using the CLI 8 Inserting a Switch to Operation Mode Using CLI 9 Configuring Physical Ports in Leaf Nodes Using the NX-OS CLI 9 Configuring Port Channels in Leaf Nodes Using the NX-OS CLI 12 Configuring Virtual Port Channels in Leaf Nodes Using the NX-OS CLI 17 Reflective Relay (802.1Qbg) 22 Enabling Reflective Relay Using the NX-OS CLI 23 Configuring Policy Groups for Interfaces 24 Configuring Overrides for Interfaces 26 About Forwarding Error Correction 28 Configuring FEC Using NX-OS Style CLI 29 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide iii Contents CHAPTER 3 Configuring APIC High Availability 31 About High Availability for APIC Cluster 31 Switching Over Active APIC with Standby APIC Using CLI 32 CHAPTER 4 Configuring Tenants 33 Creating a Tenant, VRF, and Bridge Domain 33 Additional Bridge Domain Configuration 36 Configuring an Enforced Bridge Domain 37 Configuring an Enforced Bridge Domain Using the Basic GUI 38 Configuring an Enforced Bridge Domain Using the NX-OS Style CLI 39 Creating an Application Endpoint Group 40 Configuring Legacy Forwarding Mode in the Bridge Domain 42 Configuring Contracts 44 Contract Inheritance 47 About Contract Inheritance 47 Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI 48 Configuring L2Out EPG Contract Inheritance Using the NX-OS Style CLI 52 Configuring External L3Out EPG Contract Inheritance Using the NX-OS Style CLI 54 Configuring Contract Preferred Groups 56 About Contract Preferred Groups 56 Configuring Contract Preferred Groups Using the NX-OS Style CLI 58 Exporting a Contract to Another Tenant 59 Creating Quota Management 61 About APIC Quota Management Configuration 61 Creating a Quota Management Configuration Using the NX-OS Style CLI 61 CHAPTER 5 Configuring Layer 2 External Connectivity 63 Configuring Layer 2 External Connectivity 63 Configuring VLAN Domains 66 About VLAN Domains 66 Basic VLAN Domain Configuration 67 Advanced VLAN Domain Configuration 68 Associating a VLAN Domain to a Port 70 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide iv Contents Associating a VLAN Domain to a Port-Channel 71 Associating a VLAN Domain to a Template Policy-Group 72 Associating a VLAN Domain to a Template Port-Channel 72 Associating a VLAN Domain to a Virtual Port-Channel 73 Configuring Q-in-Q Encapsulation Mapping for EPGs 74 Q-in-Q Encapsulation Mapping for EPGs 74 Mapping EPGs to Q-in-Q Encapsulated Leaf Interfaces Using the NX-OS Style CLI 75 Support Fibre Channel over Ethernet Traffic on the ACI Fabric 76 Supporting Fibre Channel over Ethernet Traffic on the ACI Fabric 76 FCoE NX-OS Style CLI Configuration 78 Configuring FCoE Connectivity Without Policies or Profiles Using the NX-OS Style CLI 78 Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS Style CLI 82 Configuring FCoE Over FEX Using NX-OS Style CLI 85 Verifying FCoE Configuration Using the NX-OS Style CLI 86 Undeploying FCoE Elements Using the NX-OS Style CLI 87 Configuring 802.1Q Tunnels 89 About ACI 802.1Q Tunnels 89 Configuring 802.1Q Tunnels Using the NX-OS Style CLI 91 Example: Configuring an 802.1Q Tunnel Using Ports with the NX-OS Style CLI 92 Example: Configuring an 802.1Q Tunnel Using Port-Channels with the NX-OS Style CLI 93 Example: Configuring an 802.1Q Tunnel Using Virtual Port-Channels with the NX-OS Style CLI 93 Configuring Dynamic Breakout Ports 94 Configuration of Dynamic Breakout Ports 94 Configuring Dynamic Breakout Ports Using the NX-OS Style CLI 95 Microsegmentation on Virtual Switches 98 Configuring Microsegmentation on Virtual Switches 98 Configuring Microsegmentation with Cisco ACI Using the NX-OS-style CLI 98 Configuring Microsegmentation on Bare-Metal 100 Using Microsegmentation with Network-based Attributes on Bare Metal 100 Configuring a Network-Based Microsegmented EPG in a Bare-Metal Environment Using the NX-OS Style CLI 100 Configuring Layer 2 IGMP Snoop Multicast 102 About Cisco APIC and IGMP Snooping 102 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide v Contents Enabling IGMP Snooping Static Port Groups 102 Configuring and Assigning an IGMP Snooping Policy to a Bridge Domain using the NX-OS Style CLI 103 Enabling IGMP Snooping and Multicast on Static Ports in the NX-OS Style CLI 104 Enabling IGMP Snoop Access Groups 105 Enabling Group Access to IGMP Snooping and Multicast using the NX-OS Style CLI 106 Deploying an EPG on a Specific Port with APIC Using the NX-OS Style CLI 108 Configuring Port Security 109 About Port Security and ACI 109 Port Security Guidelines and Restrictions 109 Port Security at Port Level 109 Configuring a Port Security Policy Group Template 110 Configuring Port Security on an Interface Using a Template 111 Configuring Port Security on an Interface Using Overrides 112 802.1x Port and Node Authentication 113 802.1x Port and Node Authentication 113 Configuring a Port Authentication Policy 113 Configuring a Node Authentication Policy 115 Configuring Proxy ARP 116 About Proxy ARP 116 Guidelines and Limitations 123 Configuring Proxy ARP Using the Cisco NX-OS Style CLI 124 CHAPTER 6 Configuring Layer 3 External Connectivity 127 About the Modes of Configuring Layer 3 External Connectivity 127 Configuring Layer 3 External Connectivity 129 Layer 3 Out to Layer 3 Out Inter-VRF Leaking 129 Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Named Example 130 Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Implicit Example 131 About SVI External Encapsulation Scope 133 Encapsulation Scope Syntax 135 Configuring SVI Interface Encapsulation Scope Using NX-OS Style CLI 136 About SVI Auto State 136 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide vi Contents Guidelines and Limitations for SVI Auto State Behavior 137 Configuring SVI Auto State Using NX-OS Style CLI 137 Configuring an Interface and Static Route 138 OSPF Configuration 141 Configuring OSPF 141 Creating OSPF VRF and Interface Templates 145 BGP Configuration 148 Configuring BGP 148 Creating BGP Address Family and Timer Templates 149 Configuring BGP Address Family and Timers 151 Configuring a BGP Neighbor 152 Configuring a Per VRF Per Node BGP Timer Policy Using the NX-OS Style CLI 156 Configuring BGP Max Path 157 Configuring AS Path Prepend 158 Configuring AS Path Prepend Using the NX-OS Style CLI 159 Route Distribution Into BGP 159 Configuring a Route-Profile with Tenant Scope 159 Configuring a Redistribute Route-Profile 160 Configuring BGP Route Dampening 161 EIGRP Configuration 164 Creating EIGRP VRF and Interface Templates 164 Configuring EIGRP Address Family and Counters 166 Configuring an EIGRP Interface 168 Configuring Route-Maps 171 Configuring Templates 171 About Route Profiles 171 Configuring a Tenant-Scoped Route Profile 171 Configuring a VRF-Scoped Route Profile 173 Creating a Route-Map 175 Configuring Route-Maps in Routing Protocols 179 Configuring an Export Map (Inter-VRF Route Leak) 180 Configuring Bi-Directional Route Forwarding (BFD) 182 About BFD 182 Configuring BFD Globally 182 Overriding Global BFD Settings 185 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide vii Contents Configuring BFD Interface Override Policy 185 Applying the BFD Interface Override Policy to Interfaces 187 Enabling BFD on Consumer Protocols 189 Enabling BFD on the BGP Consumer Protocol 189 Enabling BFD on the EIGRP Consumer Protocol 191 Enabling BFD on the OSPF Consumer Protocol 192 Enabling BFD on the Static Route Consumer Protocol 192 Configuring Layer 3 Multicast 193 Layer 3 Multicast 193 Guidelines for Configuring Layer 3 Multicast 194 Configuration Steps for Layer 3 Multicast 195 Configuring PIM Options for Layer 3 Multicast 196 Configuring IGMP Options on the VRF for Layer 3 Multicast 198 Configuring an L3 Out for Layer 3 Multicast 203 Example: Configuring Layer 3 Multicast 206 Configuring External-L3 EPGs 207 Configuring Layer 3 External Connectivity Using the Named Mode 209 Creating a Named L3Out 209 Configuring Layer 3 Interfaces for a Named L3Out 211 Configuring Route Maps for a Named L3Out 213 Configuring Routing Protocols for a Named L3Out 216 Configuring BGP for a Named L3Out 216 Configuring OSPF for a Named L3Out 218 Configuring EIGRP for a Named L3Out 220 Configuring External-L3 EPGs for a Named L3Out 222 Configuring HSRP 223 Configuring HSRP in Cisco APIC Using Inline Parameters in NX-OS Style CLI 223 Configuring HSRP in Cisco APIC Using Template and Policy in NX-OS Style CLI 224 Cisco ACI GOLF 226 Cisco ACI GOLF 226 Configuration Tasks to Configure Cisco ACI GOLF Services Using the NX-OS Style CLI 228 Configuring a Spine and the Infra Tenant for BGP EVPN, Using the NX-OS Style CLI 228 Configuring BGP to Support BGP EVPN on a Spine, Using the NX-OS Style CLI 231 Configuring a Tenant for BGP EVPN Using the NX-OS Style CLI 232 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide viii Contents Configuring a Route Map 234 Enabling Distributing BGP EVPN Type-2 Host Routes to a DCIG Using the NX-OS Style CLI 236 Cisco ACI GOLF Configuration Example, Using the NX-OS Style CLI 236 Troubleshooting EVPN Type-2 Route Distribution to a DCIG 238 Multipod_Fabric 241 About Multipod Fabric 241 Assigning Switches in a Multipod Fabric 241 Configuring Fabric-External Connectivity for a Multipod Fabric 242 Configuring Spine Interfaces and OSPF for a Multipod Fabric 244 Cisco APIC Quality of Service 247 CoS Preservation 247 Preserving 802.1P Class of Service Settings 247 Preserving QoS CoS Settings Using the NX-OS Style CLI 248 Multipod QoS 249 Enabling Multipod QoS With a DSCP Policy, Using the NX-OS Style CLI 249 Preserving QoS Priority Settings in a Multipod Fabric 250 Translating QoS Ingress Markings to Egress Markings 251 Translating QoS Ingress Markings to Egress Markings 251 Translating QoS CoS Settings Using the NX-OS CLI 251 CHAPTER 7 Configuring Management Interfaces 255 Configuring Out-of-Band Management Access 255 Configuring Inband Management Access 257 Configuring Inband Management Access to a Switch from an Outside Network 257 Configuring Inband Management Access to a Controller from an Outside Network 259 Configuring Inband Management Connectivity to the Management Station 261 Configuring Inband Management Contract to Open HTTPS/SSH Ports 263 CHAPTER 8 Configuring Security 265 About Security Configuration 265 Configuring AAA 266 Configuring Security Servers 269 Configuring a RADIUS Server 269 Configuring a TACACS+ Server 272 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide ix Contents Configuring an LDAP Server 273 Configuring the Password Policy 276 Configuring Users 279 Configuring a Locally Authenticated User 279 Configuring a Certificate and SSH-Key for a Local User 281 Configuring Public Key Infrastructure 283 Configuring a Certificate Authority and Chain of Trust 283 Configuring Keys and a Keyring 283 Generating a Certificate Signing Request 285 Configuring Webtokens 287 Configuring Communication Policies 288 Configuring the HTTP Policy 288 Configuring the HTTPS Policy 289 Configuring the SSH Policy 291 Configuring the Telnet Policy 292 Configuring AES Encryption 293 Configuring Fabric Secure Mode 294 Configuring COOP Authentication 295 About COOP Authentication 295 Configuring COOP Authentication 296 Configuring FIPS 296 About Federal Information Processing Standards (FIPS) 296 Guidelines and Limitations 297 Configuring FIPS for Cisco APIC Using NX-OS Style CLI 297 Configuring Control Plane Policing 298 Information About CoPP 298 Guidelines and Limitations for CoPP 300 Configuring CoPP Using the Cisco NX-OS CLI 300 Configuring First Hop Security 301 About First Hop Security 301 ACI FHS Deployment 301 Guidelines and Limitations 302 Configuring FHS Using the NX-OS CLI 302 CHAPTER 9 Configuring VMM 309 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide x Contents CHAPTER 10 Configuring Layer 4 - Layer 7 Services 311 CHAPTER 11 Configuring Global Policies 313 About Global Policies 313 Configuring Out-of-Band Management NTP 314 Configuring the System Clock 316 Configuring Error Disable Recovery 317 Configuring Link Level Discovery Protocol 318 Configuring Miscabling Protocol 318 Configuring the Endpoint Loop Protection Policy 320 Configuring IP Aging 321 Overview 321 Configuring the IP Aging Policy Using the NX-OS-Style CLI 321 Configuring the Dynamic Load Balancer 321 Configuring Spanning Tree Protocol 322 Configuring IS-IS 324 Configuring BGP Route Reflectors 326 Decommissioning a Node 327 Configuring Power Management 328 Configuring a Scheduler 329 Configuring System MTU 332 About PTP 332 Guidelines and Limitations 334 Configuring PTP Using the NX-OS CLI 335 CHAPTER 12 Configuring Cisco Tetration Analytics 339 Overview 339 Configuring Cisco Tetration Analytics Using the NX-OS Style CLI 339 CHAPTER 13 Configuring NetFlow 343 About NetFlow 343 Configuring a NetFlow Exporter Policy for Virtual Machine Networking Using the NX-OS-Style CLI 344 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xi Contents Configuring the NetFlow and Tetration Analytics Feature Priority Through a Node Control Policy Using the NX-OS-Style CLI 344 Configuring a NetFlow Node Policy Using the NX-OS-Style CLI 345 Configuring NetFlow Infra Selectors Using the NX-OS-Style CLI 346 Configuring NetFlow Overrides Using the NX-OS-Style CLI 348 Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style CLI 348 Consuming a NetFlow Exporter Policy Under a VMM Domain Using the NX-OS-Style CLI for VMware VDS 351 Enabling or Disabling NetFlow on an Endpoint Group Using the NX-OS-Style CLI for VMware VDS 352 CHAPTER 14 Managing Firmware 353 Managing Firmware 353 Adding or Removing Repository Images 354 Changing Catalog Firmware 354 Upgrading Controller Firmware 355 Upgrading Switch Firmware 357 CHAPTER 15 Managing the Configuration with Snapshots 361 About Configuration Management and Snapshots 361 Exporting a Snapshot 361 Importing a Snapshot 363 Rollback Configuration Using Snapshots 364 Uploading or Downloading a Snapshot File to a Remote Path 366 Managing Snapshot Files and Jobs 367 CHAPTER 16 Configuring Monitoring 369 Configuring Syslog 369 Configuring a Logging Server Group 369 Configuring Syslog 371 Configuring Call Home 372 Configuring the Call Home Policy 372 Configuring a Call Home Destination Profile 374 Call Home Destination Profile Configuration Commands 376 Configuring a Call Home Query 377 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xii Contents Query Subtree Categories 378 Sending an On-Demand Techsupport File Using the NX-OS Style CLI 380 Configuring a Remote Path for File Export 381 Using Show Commands for Monitoring 382 About Using the Show Commands 382 Using the show faults Command 382 Using the show events Command 384 Using the show health Command 385 Using the show audits Command 385 Using the show stats Command 387 Entity Filters for Show Commands 387 Configuring SNMP 388 CHAPTER 17 Configuring SPAN 391 Configuring SPAN and ERSPAN 391 Configuring Local SPAN in Access Mode 391 Configuring ERSPAN in Access Mode 393 Configuring ERSPAN in Fabric Mode 396 Configuring ERSPAN in Tenant Mode 398 CHAPTER 18 Applying the show running config Output to Another Cisco APIC 401 About Import and Export Configurations 401 Import and Export Configuration Guidelines and Limitations 401 Exporting a CLI Configuration 402 Importing a CLI Configuration 402 CHAPTER 19 Configuring a Forwarding Scale Profile Policy 405 Overview 405 Supported Platforms for the IPv4 Forwarding Scale Profile Policy 406 Configuring the Forwarding Scale Profile Policy Using the NX-OS-Style CLI 406 APPENDIX A Verified Scalability Using the CLI 409 CLI Scalability Limits 409 APPENDIX B Use Case: Three-Tier Application with Transit Topology 411 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xiii Contents About Deploying a Three-Tier Application with Transit Topology 411 Deploying a Three-Tier Application 413 Transit Routing with OSPF and BGP 415 APPENDIX C Examples: Show Commands 417 Examples: Show Commands 417 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xiv Preface • Audience, page xv • New and Changed Information, page xv • Document Conventions, page xxiii • Related Documentation, page xxiv • Documentation Feedback, page xxv Audience This guide is intended for network and systems administrators who configure and maintain the Application Centric Infrastructure fabric. New and Changed Information The following table provides an overview of the significant changes to this guide up to the current release. The table does not provide an exhaustive list of all changes made to the guide or of the new features up to this release. The APIC Release 2.3(3x) feature is only available in this specific release and no other release. Table 1: New and Changed Behavior in Cisco ACI, Release 2.3(3x) Feature Description Where Documented SVI Auto State Allows for the SVI auto state in Configuring Layer 3 External Switch Virtual Interface behavior Connectivity to be enabled. This allows the SVI state to be in the down state when all the ports in the VLAN go down. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xv Preface New and Changed Information Table 2: New and Changed Behavior in Cisco ACI, Release 3.0(1k) Feature Description Where Documented Forwarding Scale Profile Policy The forwarding scale profile policy Configuring a Forwarding Scale enables you to choose between Profile Policy Dual Stack (the default profile) and IPv4 Scale. A forwarding scale profile policy that is set to Dual Stack provides scalability of up to 6K endpoints for IPv6 configurations and up to 12K endpoints for IPv4 configurations. The IPv4 Scale option enables systems with no IPv6 configurations to increase scalability with up to 24K IPv4 endpoints. Graceful Insertion and Removal (GIR) Mode The Graceful Insertion and Removing a Switch to Maintenance Removal (GIR) mode or Mode Using the CLI maintenance mode allows you to isolate a switch from the network with minimum service disruption. Q-in-Q Encapsulation Mapping for Using Cisco APIC, you can map Configuring Q-in-Q Encapsulation EPGs double-tagged VLAN traffic Mapping for EPGs in Configuring ingressing on a regular interface, Layer 2 External Connectivity PC, or VPC to an EPG. When this feature is enabled, when double-tagged traffic enters the network for an EPG, both tags are processed individually in the fabric and restored to double-tags when egressing the ACI switch. Ingressing single-tagged and untagged traffic is dropped. With this release, you can configure an 802.1x Port Authentication policy or 802.1x Node Authentication Policy. First Hop Security Enables better IPv4 and IPv6 link Configuring First Hop Security in security and management over the Configuring Security layer 2 links. Precision Time Protocol Time synchronization protocol defined in IEEE 1588 for nodes distributed across the APIC. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xvi Configuring 802.1x Port Authentication Policy and Configuring 802.1x Node Authentication Policy in Configuring Layer 2 Connectivity 802.1x Port Authentication Configuring PTP in Configuring Global Policies Preface New and Changed Information Feature Description Where Documented Enforced Bridge Domain Enforced bridge domain is Enforced Bridge Domain in supported, in which an endpoint in Configuring Tenants a subject endpoint group (EPG) can only ping subnet gateways within the associated bridge domain. With this configuration enabled, you can create a global exception list of IP addresses which can ping any subnet gateway. Table 3: New and Changed Behavior in Cisco ACI, Release 2.3(1e) Feature Description Where Documented Cisco APIC Quota Management Creates, deletes, and updates a Creating Quota Management quota management configuration which enables the admin to limit what managed objects that can be added under a given tenant or globally across tenants. Contract Inheritance To streamline associating contracts See Contract Inheritance in to new EPGs, you can now enable Configuring Tenants an EPG to inherit all the (provided/consumed) contracts associated directly to another EPG in the same tenant. Contract inheritance can be configured for application, microsegmented, L2Out, and L3Out EPGs. Any changes you make to the EPG contract master’s contracts, are received by the inheriting EPG. 802.1Q Tunnel Enhancements Now you can configure ports on Configuring Layer 2 External core-switches for use in Dot1q Connectivity Tunnels for multiple customers. You can also define access VLANs to distinguish between customers consuming the corePorts. You can also disable MAC learning on Dot1q Tunnels. Control Plane Policing Protects the control plane and separates it from the data plane, which ensures network stability, reachability, and packet delivery. Configuring Security Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xvii Preface New and Changed Information Feature Description Where Documented Encapsulation scope for SVI across With this release you can configure See Configuring Layer 3 External Layer 3 Outside networks the encapsulation scope for SVI Connectivity across Layer 3 Outside networks. See Configuring Port Channels in Leaf Nodes Using the NX-OS CLI Symmetric Hashing Symmetric hashing is now supported on port channels. Reflective relay (802.1Qbg) Reflective relay transfers switching See Configuring Fabric and for virtual machines out of the host Interfaces server to an external network switch. It provides connectivity between VMs on the same physical server and the rest of the network. It allows policies that you configure on the Cisco APIC to apply to traffic between the VMs on the same server. Microsegmentation for virtual switches Adds content for configuring microsegment EPGs on VMware VDS, Cisco AVS, and Microsoft vSwitch. See Configuring Microsegmentation on Virtual Switches Table 4: New Features and Changed Behavior in Cisco APIC 2.2(2e) Release Feature or Change Description Per VRF per node BGP timer With this release, you can define Configuring Layer 3 External and associate BGP timers on a per Connectivity VRF per node basis. Layer 3 Out to Layer 3 Out Inter-VRF Leaking With this release, shared Layer 3 Configuring Layer 3 External Outs in different VRFs can Connectivity communicate with each other using a contract. Multiple BGP communities assigned per route prefix With this release, multiple BGP Configuring Layer 3 External communities can now be assigned Connectivity per route prefix using the BGP protocol. Apply the show running config Two new CLI commands, export command output to another Cisco config and import config, were APIC added to enable running the output for the show running-config command on another Cisco APIC. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xviii Where Documented About Import and Export Configurations in Applying the show running config Output to Another Cisco APIC Preface New and Changed Information Feature or Change Description Where Documented Name change Changed name of "Layer 3 EVPN Cisco ACI GOLF and Multipod in Services for Fabric WAN" to Configuring Layer 3 External "Cisco ACI GOLF Connectivity Table 5: New Features and Changed Behavior in Cisco APIC 2.2(1n) Release Feature Description Where Documented 802.1Q Tunnels You can configure 802.1Q tunnels Configuring 802.1Q Tunnels in to enable point-to-multi-point Configuring Layer 2 External tunneling of Ethernet frames in the Connectivity fabric, with Quality of Service (QoS) priority settings. APIC Cluster High Availability Support is added to operate the APIC High Availability APICs in a cluster in an Active/Standby mode. In an APIC cluster, the designated active APICs share the load and the designated standby APICs can act as an replacement for any of the APICs in an active cluster. Contract Preferred Groups Support is added for contract Configuring Contract Preferred preferred groups that enable greater Groups in Configuring Tenants control of communication between EPGs in a VRF. If most of the EPGs in the VRF should have open communication, but a few should only have limited communication with the other EPGs, you can configure a combination of a contract preferred group and contracts with filters to control communication precisely. Dynamic Breakout Ports Support is added for connecting a Configuring Dynamic Breakout 40 Gigabit Ethernet (GE) leaf Ports in Configuring Layer 2 switch port to 4-10GE capable External Connectivity (downlink) devices (with Cisco 40-Gigabit to 4X10-Gigabit breakout cables). FCoE over FEX You can now configure FCoE over Support Fibre Channel over FEX ports. Ethernet Traffic on the ACI Fabric Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xix Preface New and Changed Information Feature Description Where Documented HSRP Support is added for HSRP, a Configuring HSRP in Configuring protocol that provides first-hop Layer 3 External Connectivity routing redundancy for IP hosts on Ethernet networks configured with a default router IP address. NetFlow Support is added for NetFlow Configuring NetFlow technology, which provides the metering base for a key set of applications, including network traffic accounting, usage-based network billing, network planning, as well as denial of services monitoring, network monitoring, outbound marketing, and data mining for both service providers and enterprise customers. VLAN Domains Moved to Configuring Layer 2 External Connectivity Configuring VLAN Domains in Configuring Layer 2 External Connectivity Table 6: New Features and Changed Behavior in Cisco APIC 2.1(1h) Release Feature Description Where Documented IP aging In this release, the IP aging, a policy for tracking and aging unused IPs on an endpoint, is supported. Configuring IP Aging Creating a route map/profile using In this release, the explicit prefix explicit prefix list using a new list is supported through a new match type. match type that is called match route destination. Configure FIPS In this release, support for FIPS. Configuring FIPS for Cisco APIC FIPS specifies certain cryptographic algorithms as secure, and it also identifies which algorithms should be used for a module to be FIPS compliant. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xx Creating a Route Map Preface New and Changed Information Feature Description Where Documented Distribute EVPN Type-2 Host Routes In this release, for optimal traffic forwarding in an EVPN topology, you can enable fabric spines to advertise host routes using EVPN type-2 (MAC-IP) routes to the DCIG along with public BD subnets in the form of BGP EVPN type-5 (IP Prefix) routes. Enabling Distributing EVPN Type-2 Host Routes Using the NX-OS in Configuring Layer 3 EVPN Services over Fabric WAN Configure IGMP snoop layer 2 multicast support In this release, IGMP snoop support is implemented which allows a network switch to monitor IGMP traffic and filter multicasts from flooding layer 2 traffic. Among the features implemented is static port group configuration and access group configuration. Enabling IGMP Snoop Static Port Groups and Enabling IGMP Snoop Access Groups in Configuring Layer 2 IGMP Snoop Multicast Configuring network-based microsegmented EPGs in a bare-metal environment In this release you can configure Configuring Microsegmentation on microsegmented EPGs with IP Bare-Metall address attributes or MAC address attributes for physical endpoint devices. Translating QoS CoS Settings In this release, you can enable the Translating QoS CoS Settings ACI Fabric to classify the traffic Using the NX-OS CLI for devices that classify the traffic based only on the CoS value. Table 7: New Features and Changed Behavior in Cisco APIC 2.0(2f) release Feature Description Where Documented Proxy ARP Proxy ARP in Cisco ACI is added About Proxy ARP, on page 116 to enable endpoints within a network or subnet to communicate with other endpoints without knowing the real MAC address of the endpoints. Tetration Analytics Cisco Tetration Analytics agent configuration is added. Overview, on page 339 Multipod QoS Support for Preserving CoS and DSCP settings is added for Multipod topologies. Preserving QoS Priority Settings in a Multipod Fabric Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xxi Preface New and Changed Information Feature Description Where Documented Layer 3 EVPN Services Over Fabric WAN More detail was added on how to Configuration Tasks to Configure configure Layer 3 EVPN services. Cisco ACI GOLF Services Using the NX-OS Style CLI, on page 228 Release Feature Where 2.0(1) Port Security About Port Security and ACI, on page 109 2.0(1) COOP Authentication About COOP Authentication, on page 295 2.0(1) Layer 3 Multicast Layer 3 Multicast, on page 193 2.0(1) Layer 3 EVPN Services Over Fabric WAN Cisco ACI GOLF , on page 226 2.0(1) Multipod Fabric About Multipod Fabric, on page 241 2.0(1) Verified Scalability Using the CLI Verified Scalability Using the CLI, on page 409 1.2(2) BFD About BFD, on page 182 Route Summarization Configuring an EIGRP Interface, on page 168 Configuring OSPF, on page 141 1.2(1) Route Dampening Configuring Layer 3 External Connectivity, on page 127 Named Mode for configuring Layer 3 external connectivity Configuring Layer 3 External Connectivity, on page 127 IPv6 support Configuring Layer 3 External Connectivity, on page 127 Initial Release -- Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xxii Preface Document Conventions Document Conventions Command descriptions use the following conventions: Convention Description bold Bold text indicates the commands and keywords that you enter literally as shown. Italic Italic text indicates arguments for which the user supplies the values. [x] Square brackets enclose an optional element (keyword or argument). [x | y] Square brackets enclosing keywords or arguments separated by a vertical bar indicate an optional choice. {x | y} Braces enclosing keywords or arguments separated by a vertical bar indicate a required choice. [x {y | z}] Nested set of square brackets or braces indicate optional or required choices within optional or required elements. Braces and a vertical bar within square brackets indicate a required choice within an optional element. variable Indicates a variable for which you supply values, in context where italics cannot be used. string A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks. Examples use the following conventions: Convention Description screen font Terminal sessions and information the switch displays are in screen font. boldface screen font Information you must enter is in boldface screen font. italic screen font Arguments for which you supply values are in italic screen font. <> Nonprinting characters, such as passwords, are in angle brackets. [] Default responses to system prompts are in square brackets. !, # An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line. This document uses the following conventions: Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xxiii Preface Related Documentation Note Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual. Caution Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data. Warning IMPORTANT SAFETY INSTRUCTIONS This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device. SAVE THESE INSTRUCTIONS Related Documentation Cisco Application Centric Infrastructure (ACI) Documentation The ACI documentation is available at the following URL: http://www.cisco.com/c/en/us/support/ cloud-systems-management/application-policy-infrastructure-controller-apic/ tsd-products-support-series-home.html. Cisco Application Centric Infrastructure (ACI) Simulator Documentation The Cisco ACI Simulator documentation is available at http://www.cisco.com/c/en/us/support/ cloud-systems-management/application-centric-infrastructure-simulator/tsd-products-support-series-home.html. Cisco Nexus 9000 Series Switches Documentation The Cisco Nexus 9000 Series Switches documentation is available at http://www.cisco.com/c/en/us/support/ switches/nexus-9000-series-switches/tsd-products-support-series-home.html. Cisco Application Virtual Switch Documentation The Cisco Application Virtual Switch (AVS) documentation is available at http://www.cisco.com/c/en/us/ support/switches/application-virtual-switch/tsd-products-support-series-home.html. Cisco Application Centric Infrastructure (ACI) Integration with OpenStack Documentation Cisco ACI integration with OpenStack documentation is available at http://www.cisco.com/c/en/us/support/ cloud-systems-management/application-policy-infrastructure-controller-apic/ tsd-products-support-series-home.html. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xxiv Preface Documentation Feedback Documentation Feedback To provide technical feedback on this document, or to report an error or omission, please send your comments to [email protected] We appreciate your feedback. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xxv Preface Documentation Feedback Cisco APIC NX-OS Style Command-Line Interface Configuration Guide xxvi CHAPTER 1 Using the APIC CLI • Accessing the NX-OS Style CLI, page 1 • Using the NX-OS Style CLI for APIC, page 2 • Differences in Usage from NX-OS, page 4 • Mixing the NX-OS Style CLI and Basic and Advanced APIC GUI, page 5 Accessing the NX-OS Style CLI Note From Cisco APIC Release 1.0 until Release 1.2, the default CLI was a Bash shell with commands to directly operate on managed objects (MOs) and properties of the Management Information Model. Beginning with Cisco APIC Release 1.2, the default CLI is a NX-OS style CLI. The object model CLI is available by typing the bash command at the initial CLI prompt. Procedure Step 1 From a secure shell (SSH) client, open an SSH connection to APIC at [email protected] Use the administrator login name and the out-of-band management IP address that you configured during the initial setup. For example, [email protected] Step 2 When prompted, enter the administrator password. What to Do Next When you enter the NX-OS style CLI, the initial command level is the EXEC level. From this level, you can reach these configuration modes: • To continue in the NX-OS style CLI, you can stay in EXEC mode or you can type configure to enter global configuration mode. For information about NX-OS style CLI commands, see the Cisco APIC NX-OS Style CLI Command Reference. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 1 Using the APIC CLI Using the NX-OS Style CLI for APIC • To reach the object model CLI, type bash. For information about object mode CLI commands, see the Cisco APIC Command-Line Interface User Guide, APIC Releases 1.0 and 1.1. Using the NX-OS Style CLI for APIC Using CLI Command Modes The NX-OS style CLI is organized in a hierarchy of command modes with EXEC mode as the root, containing a tree of configuration submodes beginning with global configuration mode. The commands available to you depend on the mode you are in. To obtain a list of available commands in any mode, type a question mark (?) at the system prompt. This table lists and describes the two most commonly used modes (EXEC and global configuration) along with an example submode (DNS). The table shows how to enter and exit the modes, and the resulting system prompts. The system prompt helps to identify which mode you are in and the commands that are available to you in that mode. Mode Access Method Prompt EXEC From the APIC prompt, enter execsh. apic# Global configuration From EXEC mode, enter the configure command. apic(config)# DNS configuration From global configuration mode, enter the dns command. apic(config-dns)# Exit Method To exit to the login prompt, use the exit command. To exit from a configuration submode to its parent mode, use the exit command. To exit from any configuration mode or submode to EXEC mode, use the end command. CLI Command Hierarchy Configuration mode has several submodes, with commands that perform similar functions grouped under the same level. For example, all commands that display information about the system, configuration, or hardware are grouped under the show command, and all commands that allow you to configure the switch are grouped under the configure command. To execute a command that is not available in EXEC mode, you navigate to its submode starting at the top level of the hierarchy. For example, to configure DNS settings, use the configure command to enter the global configuration mode, then enter the dns command. When you are in the DNS configuration submode, you can query the available commands. as in this example: apic1# configure apic1(config)# dns apic1(config-dns)# ? address Configure the ip address for dns servers domain Configure the domains for dns servers Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 2 Using the APIC CLI Using the NX-OS Style CLI for APIC exit fabric no show use-vrf where Exit from current mode Show fabric related information Negate a command or set its defaults Show running system information Configure the management vrf for dns servers Show the current mode apic1(config-dns)# end apic1# Each submode places you further down in the prompt hierarchy. To view the hierarchy for the current mode, use the configure command, as shown in this example: apic1# configure apic1(config)# bgp-fabric apic1(config-bgp-fabric)# where configure t; bgp-fabric apic1(config-bgp-fabric)# To leave the current level and return to the previous level, type exit. To return directly to the EXEC level, type end. EXEC Mode Commands When you start a CLI session, you begin in EXEC mode. From EXEC mode, you can enter configuration mode. Most EXEC commands are one-time commands, such as show commands, which display the current configuration status. Configuration Mode Commands Configuration mode allows you to make changes to the existing configuration. When you save the configuration, these commands are saved across switch reboots. Once you are in configuration mode, you can enter a variety of protocol-specific modes. Configuration mode is the starting point for all configuration commands. Listing Commands and Syntax In any command mode, you can obtain a list of available commands by entering a question mark (?). apic1(config-dns)# ? address Configure the ip address for dns servers domain Configure the domains for dns servers exit Exit from current mode fabric Show fabric related information no Negate a command or set its defaults show Show running system information use-vrf Configure the management vrf for dns servers where Show the current mode apic1(config-dns)# end apic1# To see a list of commands that begin with a particular character sequence, type those characters followed by a question mark (?). Do not include a space before the question mark. apic1(config)# sh ? aaa Show AAA information access-list Show Access-list Information accounting Show accounting information acllog Show acllog information . . . To complete a command after you begin typing, type a tab. apic1# qu<TAB> Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 3 Using the APIC CLI Differences in Usage from NX-OS apic1# quota To list keywords or arguments, enter a question mark in place of a keyword or argument. Include a space before the question mark. This form of help is called command syntax help because it reminds you which keywords or arguments are applicable based on the commands, keywords, and arguments you have already entered. apic1(config-dns)# use-vrf ? inband-mgmt Configure dns on inband oob-mgmt Configure dns on out-of-band apic1(config-dns)# You can also abbreviate a command if the abbreviation is unambiguous. In this example, the configure command is abbreviated. apic1# conf apic1(config)# Undoing or Reverting to Default Values or Conditions Using the 'no' Prefix For many configuration commands, you can precede the command with the no keyword to remove a setting or to restore a setting to the default value. This example shows how to remove a previously-configured DNS address from the configuration. apic1(config-dns)# address 192.0.20.123 preferred apic1(config-dns)# show dns-address Address Preferred ------------------- --------192.0.20.123 yes apic1(config-dns)# no address 192.0.20.123 apic1(config-dns)# show dns-address Address Preferred ------------------- --------- Executing BASH Commands From the NX-OS Style CLI To execute a single command in the bash shell, type bash -c 'path/command' as shown in this example. apic1# bash -c '/controller/sbin/acidiag avread' You can execute a bash command from any mode or submode in the NX-OS style CLI. Entering Configuration Text with Spaces or Special Characters When a configuration field consists of user-defined text, special characters such as '$' should be escaped ('\$') or the entire word or string should be wrapped in single quotes to avoid misinterpretation by Bash. Differences in Usage from NX-OS The usage of the NX-OS style CLI for APIC differs from the traditional NX-OS CLI in these ways: • Global configuration mode is entered with the configure command instead of configure terminal. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 4 Using the APIC CLI Mixing the NX-OS Style CLI and Basic and Advanced APIC GUI • To perform node-level configuration on a particular leaf switch, you must first navigate to that switch using the leaf command. • The command syntax for specifying a physical port is slightly different. For example, an Ethernet port is specified as eth x/y instead of ethx/y. • When a configuration field consists of user-defined text, such as a password, special characters such as '$' or '!' should be escaped with a backslash ('\$') or the entire word or string should be wrapped in single quotes to avoid misinterpretation by Bash. • Some command shortcuts are different due to Bash behavior: ◦Ctrl-D exits a session. ◦Ctrl-Z suspends a job. • OSPF configuration adds area route-map and area connectivity commands. Mixing the NX-OS Style CLI and Basic and Advanced APIC GUI Basic mode will be deprecated after Cisco APIC Release 3.0(1). Cisco does not recommend using Basic mode for configuration. However, if you want to use Basic mode, use the following URL: APIC URL/indexSimple.html Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 5 Using the APIC CLI Mixing the NX-OS Style CLI and Basic and Advanced APIC GUI Caution Changes made through the APIC Basic GUI can be seen, but cannot be modified in the Advanced GUI, and changes made in the Advanced GUI cannot be rendered in the Basic GUI. The Basic GUI is kept synchronized with the NX-OS style CLI, so that if you make a change from the NX-OS style CLI, these changes are rendered in the Basic GUI, and changes made in the Basic GUI are rendered in the NX-OS style CLI, but the same synchronization does not occur between the Advanced GUI and the NX-OS style CLI. See the following examples: • Do not mix Basic and Advanced GUI modes. If you apply an interface policy to two ports using Advanced mode and then change the settings of one port using Basic mode, your changes might be applied to both ports. • Do not mix the Advanced GUI and the CLI, when doing per-interface configuration on APIC. Configurations performed in the GUI, may only partially work in the NX-OS CLI. For example, if you configure a switch port in the GUI at Tenants > tenant-name > Application Profiles > application-profile-name > Application EPGs > EPG-name > Static Ports > Deploy Static EPG on PC, VPC, or Interface Then you use the show running-config command in the NX-OS style CLI, you receive output such as: leaf 102 interface ethernet 1/15 switchport trunk allowed vlan 201 tenant t1 application ap1 epg ep1 exit exit If you use these commands to configure a static port in the NX-OS style CLI, the following error occurs: apic1(config)# leaf 102 apic1(config-leaf)# interface ethernet 1/15 apic1(config-leaf-if)# switchport trunk allowed vlan 201 tenant t1 application ap1 epg ep1 No vlan-domain associated to node 102 interface ethernet1/15 encap vlan-201 This occurs because the CLI has validations that are not performed by the APIC GUI. For the commands from the show running-config command to function in the NX-OS CLI, a vlan-domain must have been previously configured. The order of configuration is not enforced in the GUI. • Do not make changes with the Basic GUI or the NX-OS CLI before using the Advanced GUI. This may also inadvertantly cause objects to be created (with names prepended with _ui_) which cannot be changed or deleted in the Advanced GUI. For the steps to remove such objects, see Troubleshooting Unwanted _ui_ Objects in the APIC Troubleshooting Guide. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 6 CHAPTER 2 Configuring Fabric and Interfaces • Fabric and Interface Configuration, page 7 • Graceful Insertion and Removal (GIR) Mode, page 8 • Configuring Physical Ports in Leaf Nodes Using the NX-OS CLI, page 9 • Configuring Port Channels in Leaf Nodes Using the NX-OS CLI, page 12 • Configuring Virtual Port Channels in Leaf Nodes Using the NX-OS CLI, page 17 • Reflective Relay (802.1Qbg), page 22 • Configuring Policy Groups for Interfaces, page 24 • Configuring Overrides for Interfaces, page 26 • About Forwarding Error Correction, page 28 Fabric and Interface Configuration To form the ACI fabric, Cisco Nexus 9000 Series ACI-mode switches are deployed in a leaf/spine “Clos” topology managed by the APIC controller. Each leaf node is connected to all spine nodes with no connectivity between the leaf nodes. The interconnecting links between the leaf and spine nodes are called fabric links and the respective ports are called fabric ports. The fabric ports do not require user configuration for normal operation as these are auto discovered and factory default configuration is applied during fabric bring-up. All endpoint devices are connected to the leaf nodes through access ports. The access ports must be configured similar to those in NX-OS switches. Both fabric and access ports are represented as Interfaces as in NX-OS. The leaf and spine nodes are considered different objects in the ACI model and support different sets of policies. In the CLI, these nodes are represented as leaf and spine respectively while both are commonly referred to as nodes. Leaf and spine node values are unique across all the pods in the fabric. FEX modules, if attached to the leaf nodes, will have fex-id values unique only within each leaf. For example, two leaf nodes can each have a FEX 101 attached. Interface Naming In ACI fabric, most interface configuration is done for physical ports, port-channels, or vPCs (either directly connected to leaf nodes or connected through FEX modules). The general command syntax for each interface type is shown in the following table. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 7 Configuring Fabric and Interfaces Graceful Insertion and Removal (GIR) Mode Interface Type Command Syntax Examples Port interface ethernet slot/port interface eth 1/1 FEX Port interface ethernet fex-id/slot/port interface eth 101/1/1 Port-channel interface port-channel name FEX Port-channel interface port-channel name fex interface port-channel foo fex 101 fex-id Virtual Port-channel (VPC) interface vpc name vPC over FEX interface vpc name fex fex-a fex-b interface vpc foo fex 101 102 interface port-channel foo interface vpc foo Graceful Insertion and Removal (GIR) Mode The Graceful Insertion and Removal (GIR) mode or maintenance mode allows you to isolate a switch from the network with minimum service disruption. In the GIR mode you can perform real-time debugging without affecting traffic. You can use graceful insertion and removal to gracefully remove a switch and isolate it from the network in order to perform debugging operations. The switch is removed from the regular forwarding path with minimal traffic disruption. When you are finished performing the debugging operations, you can use graceful insertion to return the switch to its fully operational (normal) mode. In graceful removal, all protocols and vPC domains are gracefully brought down and the switch is isolated from the network. In addition, all the front-panel interfaces are shutdown on the switch except the fabric interfaces. In graceful insertion, all protocols and vPC domains are restored. The following protocols are supported • Border Gateway Protocol (BGP) • Enhanced Interior Gateway Routing Protocol (EIGRP) • Intermediate System-to-Intermediate System (ISIS) • Open Shortest Path First (OSPF) Important Notes • Downgrading a switch in maintenance mode is not supported. • When the switch is in maintenance mode, the Ethernet Port Module stops propagating the interface related notifications. As a result, if the remote switch is rebooted or the fabric link is flapped, the fabric link will not come up unless the switch is recommissioned. Removing a Switch to Maintenance Mode Using the CLI Use this procedure to remove a switch to maintenance mode using the CLI. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 8 Configuring Fabric and Interfaces Inserting a Switch to Operation Mode Using CLI Procedure Step 1 Command or Action Purpose [no]debug-switch node_id or node_name Removes the switch to maintenance mode. Inserting a Switch to Operation Mode Using CLI Use this procedure to insert a switch to operational mode using the CLI. Procedure Step 1 Command or Action Purpose [no]no debug-switch node_id or node_name Inserts the switch to operational mode. Configuring Physical Ports in Leaf Nodes Using the NX-OS CLI The commands in the following examples create many managed objects (MOs) in the ACI policy model that are fully compatible with the REST API/SDK and GUI. However, the CLI user can focus on the intended network configuration instead of ACI model internals. The following figure shows examples of Ethernet ports directly on leaf nodes or FEX modules attached to leaf nodes and how each is represented in the CLI. For FEX ports, the fex-id is included in the naming of the port itself as in ethernet 101/1/1. While describing an interface range, the ethernet keyword need not be repeated as in NX-OS. Example: interface ethernet 101/1/1-2, 102/1/1-2. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 9 Configuring Fabric and Interfaces Configuring Physical Ports in Leaf Nodes Using the NX-OS CLI • Leaf node ID numbers are global. • The fex-id numbers are local to each leaf. • Note the space after the keyword ethernet. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 leaf node-id Example: apic1(config)# leaf 102 Step 3 interface type Example: Specifies the leaf or leafs to be configured. The node-id can be a single node ID or a range of IDs, in the form node-id1-node-id2, to which the configuration will be applied. Specifies the interface that you are configuring. You can specify the interface type and identity. For an Ethernet port, use “ethernet slot / port.” apic1(config-leaf)# interface ethernet 1/2 Step 4 fex associate node-id Example: apic1(config-leaf-if)# fex associate 101 Step 5 speed speed Example: (Optional) If the interface or interfaces to be configured are FEX interfaces, you must use this command to attach the FEX module to a leaf node before configuration. Note This step is required before creating a port-channel using FEX ports. The speed setting is shown as an example. At this point you can configure any of the interface settings shown in the table below. apic1(config-leaf-if)# speed 10G The following table shows the interface settings that can be configured at this point. Command Purpose [no] shut Shut down physical interface [no] speed speedValue Set the speed for physical interface [no] link debounce time time Set link debounce [no] negotiate auto Configure negotiate [no] cdp enable Disable/enable Cisco Discovery Protocol (CDP) Note Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 10 CDP is not supported with FEX switches (in this case, use LLDP). Configuring Fabric and Interfaces Configuring Physical Ports in Leaf Nodes Using the NX-OS CLI Command Purpose [no] mcp enable Disable/enable Mis-cabling Protocol (MCP) [no] lldp transmit Set the transmit for physical interface [no] lldp receive Set the LLDP receive for physical interface spanning-tree {bpduguard | bpdufilter} {enable | disable} Configure spanning tree BPDU [no] storm-control level percentage [ burst-rate percentage ] Storm-control configuration (percentage) [no] storm-control pps packets-per-second burst-rate Storm-control configuration (packets-per-second) packets-per-second Examples Configure one port in a leaf node. The following example shows how to configure the interface eth1/2 in leaf 101 for the following properties: speed, cdp, and admin state. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/2 apic1(config-leaf-if)# speed 10G apic1(config-leaf-if)# cdp enable apic1(config-leaf-if)# no shut Configure multiple ports in multiple leaf nodes. The following example shows the configuration of speed for interfaces eth1/1-10 for each of the leaf nodes 101-103. apic1(config)# leaf 101-103 apic1(config-leaf)# interface eth 1/1-10 apic1(config-leaf-if)# speed 10G Attach a FEX to a leaf node. The following example shows how to attach a FEX module to a leaf node. Unlike in NX-OS, the leaf port Eth1/5 is implicitly configured as fabric port and a FEX fabric port-channel is created internally with the FEX uplink port(s). In ACI, the FEX fabric port-channels use default configuration and no user configuration is allowed. Note This step is required before creating a port-channel using FEX ports, as described in the next example. apic1(config)# leaf 102 apic1(config-leaf)# interface eth 1/5 apic1(config-leaf-if)# fex associate 101 Configure FEX ports attached to leaf nodes. This example shows configuration of speed for interfaces eth1/1-10 in FEX module 101 attached to each of the leaf nodes 102-103. The FEX ID 101 is included in the port identifier. FEX IDs start with 101 and are local to a leaf. apic1(config)# leaf 102-103 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 11 Configuring Fabric and Interfaces Configuring Port Channels in Leaf Nodes Using the NX-OS CLI apic1(config-leaf)# interface eth 101/1/1-10 apic1(config-leaf-if)# speed 1G Configuring Port Channels in Leaf Nodes Using the NX-OS CLI Port-channels are logical interfaces in NX-OS used to aggregate bandwidth for multiple physical ports and also for providing redundancy in case of link failures. In NX-OS, port-channel interfaces are identified by user-specified numbers in the range 1 to 4096 unique within a node. Port-channel interfaces are either configured explicitly (using interface port-channel command) or created implicitly (using channel-group command). The configuration of the port-channel interface is applied to all the member ports of the port-channel. There are certain compatibility parameters (speed, for example) that cannot be configured on the member ports. In the ACI model, port-channels are configured as logical entities identified by a name to represent a collection of policies that can be assigned to set of ports in one or more leaf nodes. Such assignment creates one port-channel interface in each of the leaf nodes identified by an auto-generated number in the range 1 to 4096 within the leaf node, which may be same or different among the nodes for the same port-channel name. The membership of these port-channels may be same or different as well. When port-channel is created on the FEX ports, the same port-channel name can be used to create one port-channel interface in each of the FEX attached to the leaf node. Thus, it is possible to create up to N+1 unique port-channel interfaces (identified by the auto-generated port-channel numbers) for each leaf node attached to N FEX modules. This is illustrated with the examples below. Port-channels on the FEX ports are identified by specifying the fex-id along with the port-channel name (interface port-channel foo fex 101, for example). • N+1 instances per leaf of port-channel foo are possible when each leaf is connected to N FEX nodes. • Leaf ports and FEX ports cannot be part of the same port-channel instance. • Each FEX node can have only one instance of port-channel foo. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 12 Configuring Fabric and Interfaces Configuring Port Channels in Leaf Nodes Using the NX-OS CLI Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 template port-channel channel-name Creates a new port-channel or configures an existing port-channel (global configuration). Example: apic1(config)# template port-channel foo Step 3 [no] switchport access vlan vlan-id tenant tenant-name application application-name epg epg-name Deploys the EPG with the VLAN on all ports with which the port-channel is associated. Example: apic1(config-po-ch-if)# switchport access vlan 4 tenant ExampleCorp application Web epg webEpg Step 4 channel-mode active Example: apic1(config-po-ch-if)# channel-mode active Note The channel-mode command is equivalent to the mode option in the channel-group command in NX-OS. In ACI, however, this is supported for the port-channel (not on a member port). Symmetric hashing is not supported on the following switches: Note To enable symmetric hashing, enter the lacp symmetric-hash command: apic1(config-po-ch-if)# lacp symmetric-hash • Cisco Nexus 93128TX • Cisco Nexus 9372PX • Cisco Nexus 9372PX-E • Cisco Nexus 9372TX • Cisco Nexus 9372TX-E • Cisco Nexus 9396PX • Cisco Nexus 9396TX Step 5 Returns to configure mode. exit Example: apic1(config-po-ch-if)# exit Step 6 leaf node-id Example: apic1(config)# leaf 101 Specifies the leaf switches to be configured. The node-id can be a single node ID or a range of IDs, in the form node-id1-node-id2, to which the configuration will be applied. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 13 Configuring Fabric and Interfaces Configuring Port Channels in Leaf Nodes Using the NX-OS CLI Step 7 Command or Action Purpose interface type Specifies the interface or range of interfaces that you are configuring to the port-channel. Example: apic1(config-leaf)# interface ethernet 1/1-2 Step 8 [no] channel-group channel-name Example: apic1(config-leaf-if)# channel-group foo Step 9 lacp port-priority priority Example: apic1(config-leaf-if)# lacp port-priority 1000 apic1(config-leaf-if)# lacp rate fast Assigns the interface or range of interfaces to the port-channel. Use the keyword no to remove the interface from the port-channel. To change the port-channel assignment on an interface, you can enter the channel-group command without first removing the interface from the previous port-channel. (Optional) This setting and other per-port LACP properties can be applied to member ports of a port-channel at this point. Note In the ACI model, these commands are allowed only after the ports are member of a port channel. If a port is removed from a port channel, configuration of these per-port properties are removed as well. The following table shows various commands for global configurations of port channel properties in the ACI model. These commands can also be used for configuring overrides for port channels in a specific leaf in the (config-leaf-if) CLI mode. The configuration made on the port-channel is applied to all member ports. CLI Syntax Feature [no] speed <speedValue> Set the speed for port-channel [no] link debounce time <time> Set Link Debounce for port-channel [no] negotiate auto Configure Negotiate for port-channel [no] cdp enable Disable/Enable cdp for port-channel Note [no] mcp enable Disable/Enable mcp for port-channel [no] lldp transmit Set the transmit for port-channel [no] lldp receive Set the lldp receive for port-channel spanning-tree <bpduguard | bpdufilter> <enable | disable> Configure spanning tree bpdu Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 14 CDP is not supported with FEX switches (in this case, use LLDP). Configuring Fabric and Interfaces Configuring Port Channels in Leaf Nodes Using the NX-OS CLI CLI Syntax Feature [no] storm-control level <percentage> [ burst-rate <percentage> ] Storm-control configuration (percentage) [no] storm-control pps <packet-per-second> burst-rate Storm-control configuration (packets-per-second) <packets-per-second> [no] channel-mode { active | passive | on| mac-pinning LACP mode for the link in port-channel l } [no] lacp min-links <value> Set minimum number of links [no] lacp max-links <value> Set maximum number of links [no] lacp fast-select-hot-standby LACP fast select for hot standby ports [no] lacp graceful-convergence LACP graceful convergence [no] lacp load-defer LACP load defer member ports [no] lacp suspend-individual LACP individual Port suspension [no] lacp port-priority LACP port priority [no] lacp rate LACP rate Examples Configure a port channel (global configuration). A logical entity foo is created that represents a collection of policies with two configurations: speed and channel mode. More properties can be configured as required. Note The channel mode command is equivalent to the mode option in the channel group command in NX-OS. In ACI, however, this supported for the port-channel (not on member port). apic1(config)# template apic1(config-po-ch-if)# webEpg apic1(config-po-ch-if)# apic1(config-po-ch-if)# port-channel foo switchport access vlan 4 tenant ExampleCorp application Web epg speed 10G channel-mode active Configure ports to a port-channel in a FEX. In this example, port channel foo is assigned to ports Ethernet 1/1-2 in FEX 101 attached to leaf node 102 to create an instance of port channel foo. The leaf node will auto-generate a number, say 1002 to identify the port channel in the switch. This port channel number would be unique to the leaf node 102 regardless of how many instance of port channel foo are created. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 15 Configuring Fabric and Interfaces Configuring Port Channels in Leaf Nodes Using the NX-OS CLI Note The configuration to attach the FEX module to the leaf node must be done before creating port channels using FEX ports. apic1(config)# leaf 102 apic1(config-leaf)# interface ethernet 101/1/1-2 apic1(config-leaf-if)# channel-group foo In Leaf 102, this port channel interface can be referred to as interface port-channel foo FEX 101. apic1(config)# leaf 102 apic1(config-leaf)# interface port-channel foo fex 101 apic1(config-leaf)# shut Configure ports to a port channel in multiple leaf nodes. In this example, port channel foo is assigned to ports Ethernet 1/1-2 in each of the leaf nodes 101-103. The leaf nodes will auto generate a number unique in each node (which may be same or different among nodes) to represent the port-channel interfaces. apic1(config)# leaf 101-103 apic1(config-leaf)# interface ethernet 1/1-2 apic1(config-leaf-if)# channel-group foo Add members to port channels. This example would add two members eth1/3-4 to the port-channel in each leaf node, so that port-channel foo in each node would have members eth 1/1-4. apic1(config)# leaf 101-103 apic1(config-leaf)# interface ethernet 1/3-4 apic1(config-leaf-if)# channel-group foo Remove members from port channels. This example would remove two members eth1/2, eth1/4 from the port channel foo in each leaf node, so that port channel foo in each node would have members eth 1/1, eth1/3. apic1(config)# leaf 101-103 apic1(config-leaf)# interface eth 1/2,1/4 apic1(config-leaf-if)# no channel-group foo Configure port-channel with different members in multiple leaf nodes. This example shows how to use the same port-channel foo policies to create a port-channel interface in multiple leaf nodes with different member ports in each leaf. The port-channel numbers in the leaf nodes may be same or different for the same port-channel foo. In the CLI, however, the configuration will be referred as interface port-channel foo. If the port-channel is configured for the FEX ports, it would be referred to as interface port-channel foo fex <fex-id>. apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/1-2 apic1(config-leaf-if)# channel-group foo apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# leaf 102 apic1(config-leaf)# interface ethernet 1/3-4 apic1(config-leaf-if)# channel-group foo apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# leaf 103 apic1(config-leaf)# interface ethernet 1/5-8 apic1(config-leaf-if)# channel-group foo apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 101/1/1-2 apic1(config-leaf-if)# channel-group foo Configure per port properties for LACP. This example shows how to configure member ports of a port-channel for per-port properties for LACP. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 16 Configuring Fabric and Interfaces Configuring Virtual Port Channels in Leaf Nodes Using the NX-OS CLI Note In ACI model, these commands are allowed only after the ports are member of a port channel. If a port is removed from a port channel, configuration of these per-port properties would be removed as well. apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/1-2 apic1(config-leaf-if)# channel-group foo apic1(config-leaf-if)# lacp port-priority 1000 apic1(config-leaf-if)# lacp rate fast Configure admin state for port channels. In this example, a port-channel foo is configured in each of the leaf nodes 101-103 using the channel-group command. The admin state of port-channel(s) can be configured in each leaf using the port-channel interface. In ACI model, the admin state of the port-channel cannot be configured in the global scope. // create port-channel foo in each leaf apic1(config)# leaf 101-103 apic1(config-leaf)# interface ethernet 1/3-4 apic1(config-leaf-if)# channel-group foo // configure admin state in specific leaf apic1(config)# leaf 101 apic1(config-leaf)# interface port-channel foo apic1(config-leaf-if)# shut Override config is very helpful to assign specific vlan-domain, for example, to the port-channel interfaces in each leaf while sharing other properties. // configure a port channel global config apic1(config)# interface port-channel foo apic1(config-if)# speed 1G apic1(config-if)# channel-mode active // create port-channel foo in each leaf apic1(config)# leaf 101-103 apic1(config-leaf)# interface ethernet 1/1-2 apic1(config-leaf-if)# channel-group foo // override port-channel foo in leaf 102 apic1(config)# leaf 102 apic1(config-leaf)# interface port-channel foo apic1(config-leaf-if)# speed 10G apic1(config-leaf-if)# channel-mode on apic1(config-leaf-if)# vlan-domain dom-foo This example shows how to change port channel assignment for ports using the channel-group command. There is no need to remove port channel membership before assigning to other port channel. apic1(config)# leaf 101-103 apic1(config-leaf)# interface ethernet 1/3-4 apic1(config-leaf-if)# channel-group foo apic1(config-leaf-if)# channel-group bar Configuring Virtual Port Channels in Leaf Nodes Using the NX-OS CLI A Virtual Port Channel (VPC) is an enhancement to port-channels that allows connection of a host or switch to two upstream leaf nodes to improve bandwidth utilization and availability. In NX-OS, VPC configuration Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 17 Configuring Fabric and Interfaces Configuring Virtual Port Channels in Leaf Nodes Using the NX-OS CLI is done in each of the two upstream switches and configuration is synchronized using peer link between the switches. Note When creating a VPC domain between two leaf switches, both switches must be in the same switch generation, one of the following: • Generation 1 - Cisco Nexus N9K switches without “EX” on the end of the switch name; for example, N9K-9312TX • Generation 2 – Cisco Nexus N9K switches with “EX” on the end of the switch model name; for example, N9K-93108TC-EX Switches such as these two are not compatible VPC peers. Instead, use switches of the same generation. The ACI model does not require a peer link and VPC configuration can be done globally for both the upstream leaf nodes. A global configuration mode called vpc context is introduced in ACI and VPC interfaces are represented using a type interface vpc that allows global configuration applicable to both leaf nodes. Two different topologies are supported for VPC in the ACI model: VPC using leaf ports and VPC over FEX ports. It is possible to create many VPC interfaces between a pair of leaf nodes and similarly, many VPC interfaces can be created between a pair of FEX modules attached to the leaf node pairs in a straight-through topology. VPC considerations include: • The VPC name used is unique between leaf node pairs. For example, only one VPC 'corp' can be created per leaf pair (with or without FEX). • Leaf ports and FEX ports cannot be part of the same VPC. • Each FEX module can be part of only one instance of VPC corp. • VPC context allows configuration • The VPC context mode allows configuration of all VPCs for a given leaf pair. For VPC over FEX, the fex-id pairs must be specified either for the VPC context or along with the VPC interface, as shown in the following two alternative examples. (config)# vpc context leaf 101 102 (config-vpc)# interface vpc Reg fex 101 101 or (config)# vpc context leaf 101 102 fex 101 101 (config-vpc)# interface vpc Reg In the ACI model, VPC configuration is done in the following steps (as shown in the examples below). Note A VLAN domain is required with a VLAN range. It must be associated with the port-channel template. 1 VLAN domain configuration (global config) with VLAN range 2 VPC domain configuration (global config) Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 18 Configuring Fabric and Interfaces Configuring Virtual Port Channels in Leaf Nodes Using the NX-OS CLI 3 Port-channel template configuration (global config) 4 Associate the port-channel template with the VLAN domain 5 Port-channel configuration for VPC (global config) 6 Configure ports to VPC in leaf nodes 7 Configure L2, L3 for VPC in the vpc context Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 vlan-domainname[dynamic] [type domain-type] Configures a VLAN domain for the virtual port-channel (here with a port-channel template). Example: apic1(config)# vlan-domain dom1 dynamic Step 3 Configures a VLAN range for the VLAN domain and exits the configuration mode. The range can be a single VLAN or a range of VLANs. vlanrange Example: apic1(config-vlan)# vlan 1000-1999 apic1(config-vlan)# exit Step 4 vpc domain explicit domain-id leaf node-id1 node-id2 Configures a VPC domain between a pair of leaf nodes. You can specify the VPC domain ID in the explicit mode along with the leaf node pairs. Example: Alternative commands to configure a VPC domain are as follows: apic1(config)# vpc domain explicit 1 leaf 101 102 • vpc domain [consecutive | reciprocal] The consecutive and reciprocal options allow auto configuration of a VPC domain across all leaf nodes in the ACI fabric. • vpc domain consecutive domain-start leaf start-node end-node This command configures a VPC domain consecutively for a selected set of leaf node pairs. Step 5 peer-dead-interval interval Example: apic1(config-vpc)# peer-dead-interval 10 Sets the interval between hello packets from a neighbor before the router declares the neighbor as down. The range of valid values is 5 to 600 seconds. The value must be the same for all networking devices on a specific network. Specifying a smaller dead interval (seconds) Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 19 Configuring Fabric and Interfaces Configuring Virtual Port Channels in Leaf Nodes Using the NX-OS CLI Command or Action Purpose will give faster detection of a neighbor being down and improve convergence, but might cause more routing instability. Step 6 exit Returns to global configuration mode. Example: apic1(config-vpc)# exit Step 7 template port-channel channel-name Creates a new port-channel or configures an existing port-channel (global configuration). Example: apic1(config)# template port-channel corp Step 8 vlan-domain membervlan-domain-name All VPCs are configured as port-channels in each leaf pair. The same port-channel name must be used in a leaf pair for the same VPC. This port-channel can be used to create a VPC among one or more pairs of leaf nodes. Each leaf node will have only one instance of this VPC. Associates the port channel template with the previously configured VLAN domain. Example: vlan-domain member dom1 Step 9 switchport access vlan vlan-id tenant Deploys the EPG with the VLAN on all ports with which the port-channel is associated. tenant-name application application-name epg epg-name Example: apic1(config-po-ch-if)# switchport access vlan 4 tenant ExampleCorp application Web epg webEpg Step 10 channel-mode active Note A port-channel must be in active channel-mode for a VPC. Example: apic1(config-po-ch-if)# channel-mode active Step 11 exit Returns to configure mode. Example: apic1(config-po-ch-if)# exit Step 12 leaf node-id1 node-id2 Example: apic1(config)# leaf 101-102 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 20 Specifies the pair of leaf switches to be configured. Configuring Fabric and Interfaces Configuring Virtual Port Channels in Leaf Nodes Using the NX-OS CLI Step 13 Command or Action Purpose interface typeleaf/interface-range Specifies the interface or range of interfaces that you are configuring to the port-channel. Example: apic1(config-leaf)# interface ethernet 1/3-4 Step 14 [no] channel-group channel-name vpc Assigns the interface or range of interfaces to the port-channel. Use the keyword no to remove the interface from the port-channel. To change the Example: apic1(config-leaf-if)# port-channel assignment on an interface, you can enter channel-group corp vpc the channel-group command without first removing the interface from the previous port-channel. Note Step 15 The vpc keyword in this command makes the port-channel a VPC. If the VPC does not already exist, a VPC ID is automatically generated and is applied to all member leaf nodes. exit Example: apic1(config-leaf-if)# exit Step 16 exit Example: apic1(config-leaf)# exit Step 17 vpc context leaf node-id1 node-id2 The vpc context mode allows configuration of VPC to be applied to both leaf node pairs. Example: apic1(config)# vpc context leaf 101 102 Step 18 interface vpc channel-name Example: apic1(config-vpc)# interface vpc blue fex 102 102 Step 19 [no] shutdown Example: apic1(config-vpc-if)# no shut (Optional) Administrative state configuration in the vpc context allows changing the admin state of a VPC with one command for both leaf nodes. This example shows how to configure a basic VPC. apic1# configure apic1(config)# vlan-domain dom1 dynamic apic1(config-vlan)# vlan 1000-1999 apic1(config-vlan)# exit apic1(config)# vpc domain explicit 1 leaf 101 102 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 21 Configuring Fabric and Interfaces Reflective Relay (802.1Qbg) apic1(config-vpc)# peer-dead-interval 10 apic1(config-vpc)# exit apic1(config)# template port-channel corp apic1(config-po-ch-if)# vlan-domain member dom1 apic1(config-po-ch-if)# channel-mode active apic1(config-po-ch-if)# exit apic1(config)# leaf 101-102 apic1(config-leaf)# interface ethernet 1/3-4 apic1(config-leaf-if)# channel-group corp vpc apic1(config-leaf-if)# exit apic1(config)# vpc context leaf 101 102 This example shows how to configure VPCs with FEX ports. apic1(config-leaf)# interface ethernet 101/1/1-2 apic1(config-leaf-if)# channel-group Reg vpc apic1(config)# vpc context leaf 101 102 apic1(config-vpc)# interface vpc corp apic1(config-vpc-if)# exit apic1(config-vpc)# interface vpc red fex 101 101 apic1(config-vpc-if)# switchport apic1(config-vpc-if)# exit apic1(config-vpc)# interface vpc blue fex 102 102 apic1(config-vpc-if)# shut Reflective Relay (802.1Qbg) Reflective relay is a switching option beginning with Cisco APIC Release 2.3(1). Reflective relay—the tagless approach of IEEE standard 802.1Qbg—forwards all traffic to an external switch, which then applies policy and sends the traffic back to the destination or target VM on the server as needed. There is no local switching. For broadcast or multicast traffic, reflective relay provides packet replication to each VM locally on the server. One benefit of reflective relay is that it leverages the external switch for switching features and management capabilities, freeing server resources to support the VMs. Reflective relay also allows policies that you configure on the Cisco APIC to apply to traffic between the VMs on the same server. In the Cisco ACI, you can enable reflective relay, which allows traffic to turn back out of the same port it came in on. You can enable reflective relay on individual ports, port channels, or virtual port channels as a Layer 2 interface policy using the APIC GUI, NX-OS CLI, or REST API. It is disabled by default. The term Virtual Ethernet Port Aggregator (VEPA) is also used to describe 802.1Qbg functionality. Reflective Relay Support Reflective relay supports the following: • IEEE standard 802.1Qbg tagless approach, known as reflective relay. Cisco APIC Release 2.3(1) release does not support the IEE standard 802.1Qbg S-tagged approach with multichannel technology. • Physical domains. Virtual domains are not supported. • Physical ports, port channels (PCs), and virtual port channels (VPCs). Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 22 Configuring Fabric and Interfaces Enabling Reflective Relay Using the NX-OS CLI Cisco Fabric Extender (FEX) and blade servers are not supported. If reflective relay is enabled on an unsupported interface, a fault is raised, and the last valid configuration is retained. Disabling reflective relay on the port clears the fault. • Cisco Nexus 9000 series switches with EX or FX at the end of their model name. Enabling Reflective Relay Using the NX-OS CLI Reflective relay is disabled by default; however, you can enable it on a port, port channel, or virtual port channel as a Layer 2 interface policy on the switch. In the NX-OS CLI, you can use a template to enable reflective relay on multiple ports or you can enable it on individual ports. Before You Begin This procedure assumes that you have set up the Cisco Application Centric Infrastructure (ACI) fabric and installed the physical switches. Procedure Enable reflective relay on one or multiple ports: Example: This example enables reflective relay on a single port: apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/2 apic1(config-leaf-if)# switchport vepa enabled apic1(config-leaf-if)# exit apic1(config-leaf)# exit Example: This example enables reflective relay on multiple ports using a template: apic1(config)# template policy-group grp1 apic1(config-pol-grp-if)# switchport vepa enabled apic1(config-pol-grp-if)# exit apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/2-4 apic1(config-leaf-if)# policy-group grp1 Example: This example enables reflective relay on a port channel: apic1(config)# leaf 101 apic1(config-leaf)# interface port-channel po2 apic1(config-leaf-if)# switchport vepa enabled apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# Example: This example enables reflective relay on multiple port channels: apic1(config)# template port-channel po1 apic1(config-if)# switchport vepa enabled apic1(config-if)# exit apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/3-4 apic1(config-leaf-if)# channel-group po1 apic1(config-leaf-if)# exit apic1(config-leaf)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 23 Configuring Fabric and Interfaces Configuring Policy Groups for Interfaces Example: This example enables reflective relay on a virtual port channel: apic1(config)# vpc domain explicit 1 leaf 101 102 apic1(config-vpc)# exit apic1(config)# template port-channel po4 apic1(config-if)# exit apic1(config)# leaf 101-102 apic1(config-leaf)# interface eth 1/11-12 apic1(config-leaf-if)# channel-group po4 vpc apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# vpc context leaf 101 102 apic1(config-vpc)# interface vpc po4 apic1(config-vpc-if)# switchport vepa enabled Configuring Policy Groups for Interfaces In data center networks, oftentimes configuration of many interfaces is the same across multiple nodes. This can be achieved in the ACI Policy Model by creating policy-groups to be shared by groups of interfaces across multiple leaf nodes. The policy-group is identified by a name similar to the port-channel; however, in case of port-channel the policies shared with the group of ports create one logical interface in each leaf while in case of policy-group, each of the ports sharing the policies are individual physical interfaces. The policy-group concept is very similar to port-profile in NX-OS. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 template policy-group policy-group-name Example: apic1(config)# template policy-group pg1 Step 3 [no] switchport access vlan vlan-id tenant tenant-name application application-name epg epg-name Example: apic1(config-pol-grp-if)# switchport access vlan 4 tenant ExampleCorp application Web epg webEpg Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 24 Creates a new policy group or edits an existing policy group. Configuring Fabric and Interfaces Configuring Policy Groups for Interfaces Step 4 Command or Action Purpose (Apply configuration commands) The table at the end of these steps shows various commands for configurations of policy-group for interfaces. Example: apic1(config-pol-grp-if)# speed 10G apic1(config-pol-grp-if)# cdp enable Step 5 Returns to configure mode. exit Example: apic1(config-pol-grp-if)# exit Step 6 leaf node-id Example: apic1(config)# leaf 101-103 Step 7 interface type Specifies the leaf or leafs to be configured. The node-id can be a single node ID or a range of IDs, in the form node-id1-node-id2, to which the configuration will be applied. Specifies the interface or range of interfaces to which you will apply the policy group. Example: apic1(config-leaf)# interface ethernet 1/1-24 Step 8 [no] policy-group policy-group-name Applies the policy-group to the interface or range of interfaces. Use the keyword no to remove the [force] policy-group from the interface. Use the keyword force to delete any override configurations on the interfaces. Example: apic1(config-leaf-if)# policy-group pg1 If the specified policy-group was not configured prior to this command, this command would not implicitly create the policy-group. However, the policy-group would take effect on the interface after the policy-group has been configured in the global scope. To change the policy-group assignment on an interface, you can enter the policy-group command without first removing the previous policy-group from the interface. Note If you apply a policy-group to an interface and then assign the interface to a port-channel, the interface will lose the policy-group configuration and the policies in the port-channel will be applied. The following table shows various commands for configurations of policy-group for interfaces. CLI Syntax Feature [no] speed <speedValue> Set the speed for Physical Interface [no] link debounce time <time> Set link debounce for Physical Interface Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 25 Configuring Fabric and Interfaces Configuring Overrides for Interfaces CLI Syntax Feature [no] negotiate auto Configure Negotiate for Physical Interface [no] cdp enable Disable/Enable CDP for Physical Interface Note CDP policies are not supported in interface policy groups used with FEX interfaces (in this case, use an LLDP policy). [no] mcp enable Disable/Enable MCP for Physical Interface [no] lldp transmit Set the LLDP transmit for Physical Interface [no] lldp receive Set the LLDP receive for Physical Interface spanning-tree <bpduguard | bpdufilter> <enable | disable> Configure spanning tree BPDU [no] storm-control level <percentage> [ burst-rate <percentage> ] Storm-control configuration (percentage) [no] storm-control pps <packet-per-second> burst-rate Storm-control configuration (packets-per-second) <packets-per-second> This example shows how to configure a policy-group and apply it to a range of ports in each of the leaf nodes 101-103. Each of the ports sharing the policy-group in each leaf will have the same configuration as defined in the policy-group pg1. apic1# configure apic1(config)# template policy-group pg1 apic1(config-pol-grp-if)# switchport access vlan 4 tenant ExampleCorp application Web epg webEpg apic1(config-pol-grp-if)# speed 10G apic1(config-pol-grp-if)# cdp enable apic1(config-pol-grp-if)# exit apic1(config)# leaf 101-103 apic1(config-leaf)# interface ethernet 1/1-24 apic1(config-leaf-if)# policy-group pg1 Configuring Overrides for Interfaces When policy-groups are used with large number of interfaces, it may be useful to have the option to configure a set of ports for specific properties that will override the configuration in the assigned policy-group. Override configuration is allowed only if the port is assigned to a policy-group. Override configuration is not allowed for member ports of a port-channel. When a port is added to a port-channel, the override configuration is automatically removed. However, during policy-group assignment to a port that has overrides configured, the override configuration is not removed automatically and the user can decide to remove the override configuration with the force option, if required, in the policy-group command. When a policy-group assignment is removed from a port, the override config, if exists, does not change. Similarly, the override config does not change if the port is assigned to a different policy-group (without the Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 26 Configuring Fabric and Interfaces Configuring Overrides for Interfaces force option). The override config takes effect once configured and it is not removed even if the user assigns default values to all the properties in the override. To remove the override config, the user can reapply the policy-group assignment with force option. The force option, however, is not displayed in the show running-config as it is used to just remove the override config in the ACI model. In the ACI model, overrides can be configured for a policy which may contain one or more properties. If a policy has more than one property, it is not possible to override only one property within a policy. In the CLI framework, when the user intends to override a property for which the corresponding policy has more than one property, all other properties in the policy except the override property would be implicitly copied to the override configuration to avoid ambiguity. Such implicit copy of configuration would be reflected in the output of show running-config regardless of the value (including default values). Also, the copy is done only once during the configuration of the override policy and any subsequent change to the policy-group for any of the properties in that policy would have no effect on the port(s) on which the override is configured. If the policy-group assigned to a port is not configured when the override is created, the implicit copy of properties noted above is not possible; instead, default values are assigned to properties in the override config for which the corresponding policy has more than one property. These properties shall not change for the override config when the policy-group is configured afterwards. It is recommended that user create overrides after configuring the policy-group itself or the user may need to configure the overrides in addition to the config in policy-group to get desired configuration if the config for properties in override are set to default implicitly before the configuration of the policy-group with non-default values for those properties. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 leaf node-id Example: apic1(config)# leaf 102 Step 3 interface type Specifies the leaf or leafs to be configured. The node-id can be a single node ID or a range of IDs, in the form node-id1-node-id2, to which the configuration will be applied. Specifies the interface or range of interfaces with an override configuration. Example: apic1(config-leaf)# interface ethernet 1/2 Step 4 policy-group policy-group-name force Forces the policy-group to the interface or range of interfaces, deleting any override configurations on the interfaces. Example: apic1(config-leaf-if)# policy-group pg1 force Examples This example shows how to apply a policy-group and then override the speed configuration for port eth1/1 in leaf node 101. In the ACI model, speed is part of a policy that also contains properties autoneg and link Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 27 Configuring Fabric and Interfaces About Forwarding Error Correction debounce time. As a result, those properties are copied from the speed policy-group when the override of pg1 is configured. apic1# configure apic1(config)# interface policy-group pg1 apic1(config-pol-grp-if)# speed 10G apic1(config-pol-grp-if)# cdp enable apic1(config-pol-grp-if)# exit apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/1-2 apic1(config-leaf-if)# policy-group pg1 apic1(config-pol-grp-if)# exit apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/1 apic1(config-leaf-if)# speed 1G apic1(config-leaf-if)# show running-config leaf 101 interface ethernet 1/1 policy-group pg1 speed 1G autoneg on link debounce time 100 interface ethernet 1/2 policy-group pg1 This example shows how to remove the override configuration from port eth1/1 in leaf node 101. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/1 apic1(config-leaf-if)# policy-group pg1 force apic1(config-leaf-if)# show running-config leaf 101 interface ethernet 1/1 policy-group pg1 About Forwarding Error Correction Forwarding Error Correction (FEC) is a method of obtaining error control in data transmission over an unreliable or noisy channel in which the source (transmitter) encodes the data in a redundant way using Error Correcting Code, and the destination (receiver) recognizes it and corrects the errors without requiring a retransmission. The available options are as follows: • CL74-FC-FEC—Supports 25 Gbps speed. • CL91-RS-FEC—Supports 25 and 100 Gbps speeds. • Disable-FEC—Disables FEC. • Inherit—The switch uses FEC based on the port transceiver type. All copper (CR4) transceivers have FC-FEC enabled on 25G. All interfaces with 100G transceivers have RS-FEC enabled. The default is "Inherit". Note FEC is only configurable on the front port and not on fabric ports. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 28 Configuring Fabric and Interfaces Configuring FEC Using NX-OS Style CLI Configuring FEC Using NX-OS Style CLI Procedure Step 1 Command or Action Purpose Enter the configure mode. Enters the configuration mode. Example: apic1# configure Step 2 Enter the switch mode. Enters the switch mode. Example: apic1(config)# leaf 104 Step 3 Specify the interface and port. Specifies the interface and port. Example: apic1(config-leaf)# int eth 1/4 Step 4 Configure FEC. Configures RS-FEC. Note Example: apic1(config-leaf-if)# forward-error-correction cl91-rs-fec Step 5 Exit the interface mode. The default forward-error-correction value is inherit. Exits the interface mode. Example: apic1(config-leaf-if)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 29 Configuring Fabric and Interfaces Configuring FEC Using NX-OS Style CLI Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 30 CHAPTER 3 Configuring APIC High Availability • About High Availability for APIC Cluster, page 31 • Switching Over Active APIC with Standby APIC Using CLI, page 32 About High Availability for APIC Cluster The High Availability functionality for an APIC cluster enables you to operate the APICs in a cluster in an Active/Standby mode. In an APIC cluster, the designated active APICs share the load and the designated standby APICs can act as a replacement for any of the APICs in an active cluster. As an admin user, you can set up the High Availability functionality when the APIC is launched for the first time. We recommend that you have at least three active APICs in a cluster, and one or more standby APICs. As an admin user, you can initiate the switch over to replace an active APIC with a standby APIC. Important Notes • The standby APIC is automatically updated with firmware updates to keep the backup APIC at same firmware version as the active cluster. • During an upgrade process, once all the active APICs are upgraded, the standby APIC is also be upgraded automatically. • Temporary IDs are assigned to standby APICs. After a standby APIC is switched over to an active APIC, a new ID is assigned. • Admin login is not enabled on standby APIC. To troubleshoot HA, you must log in to the standby using SSH as rescue-user. • During switch over the replaced active APIC is powered down, to prevent connectivity to the replaced APIC. • Switch over fails under the following conditions: ◦If there is no connectivity to the standby APIC. ◦If the firmware version of the standby APIC is not the same as that of the active cluster. • After switching over a standby APIC to active, if it was the only standby, you must configure a new standby. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 31 Configuring APIC High Availability Switching Over Active APIC with Standby APIC Using CLI • The following limitations are observed for retaining out of band address for standby APIC after a fail over. ◦Standby(new active) APIC may not retain its out of band address if more than 1 active APICs are down or unavailable. ◦Standby(new active) APIC may not retain its out of band address if it is in a different subnet than active APIC. ◦Standby(new active) APIC may not retain its IPv6 out of band address. Note In case you observe any of the limitations, in order to retain standby APICs out of band address, you must manually change the OOB policy for replaced APIC after the replace operation is completed successfully. • We recommend keeping standby APICs in same POD as the active APICs it may replace. • There must be three active APICs in order to add a standby APIC. • The standby APIC does not participate in policy configuration or management. • No information is replicated to standby controllers, including admin credentials. Switching Over Active APIC with Standby APIC Using CLI Use this procedure to switch over an active APIC with a standby APIC. Procedure Step 1 Command or Action Purpose replace-controller replace ID number Backup serial number Replaces an active APIC with an standby APIC. Example: apic1#replace-controller replace 2 FCH1804V27L Do you want to replace APIC 2 with a backup? (Y/n): Y Step 2 replace-controller reset ID number Example: apic1# replace-controller reset 2 Do you want to reset failover status of APIC 2? (Y/n): Y Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 32 Resets fail over status of the active controller. CHAPTER 4 Configuring Tenants • Creating a Tenant, VRF, and Bridge Domain, page 33 • Additional Bridge Domain Configuration, page 36 • Configuring an Enforced Bridge Domain, page 37 • Creating an Application Endpoint Group, page 40 • Configuring Legacy Forwarding Mode in the Bridge Domain, page 42 • Configuring Contracts, page 44 • Contract Inheritance, page 47 • Configuring Contract Preferred Groups, page 56 • Exporting a Contract to Another Tenant, page 59 • Creating Quota Management, page 61 Creating a Tenant, VRF, and Bridge Domain This topic describes the following steps in the basic provisioning of a new tenant: 1 Create a tenant 2 Associate the tenant with a security domain 3 Create a VRF for the tenant 4 Create a bridge domain for endpoint groups within the tenant Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 33 Configuring Tenants Creating a Tenant, VRF, and Bridge Domain Step 2 Command or Action Purpose tenant tenant-name Creates a tenant if it does not exist and enters the tenant configuration mode. Example: apic1(config)# tenant exampleCorp Step 3 security domain domain-name Associates the tenant with one or more security domains. Example: apic1(config-tenant)# security domain exampleCorp_dom1 Step 4 [no] vrf context vrf-name Creates a private network (VRF) for the tenant. A tenant can have one or more VRFs configured. Example: apic1(config-tenant)# vrf context exampleCorp_v1 Step 5 [no] contract {provider | consumer} contract-name Provide or consume contracts for all the EPGs under the VRF. Example: apic1(config-tenant-vrf)# contract provider web Step 6 exit Returns to the tenant configuration mode. Example: apic1(config-tenant-vrf)# exit Step 7 [no] bridge-domain bd-name Creates or deletes a bridge domain under the tenant. Enters bridge domain configuration mode. Example: apic1(config-tenant)# bridge-domain exampleCorp_b1 Step 8 [no] vrf member vrf-name Assigns the bridge-domain to a VRF. Example: apic1(config-tenant-bd)# vrf member exampleCorp_v1 Step 9 exit Returns to the tenant configuration mode. Example: apic1(config-tenant-bd)# exit Step 10 interface bridge-domain bd-name Example: apic1(config-tenant)# interface bridge-domain exampleCorp_b1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 34 Enters tenant interface configuration mode to enable routing and to apply interfaces to the bridge domain. Configuring Tenants Creating a Tenant, VRF, and Bridge Domain Command or Action Step 11 Purpose [no] {ip | ipv6} address address/mask-length Assigns or removes the gateway IP address of the bridge domain and enters the IP address mode to [scope {private | public}] [secondary] configure optional IP address properties. Example: apic1(config-tenant-if)# ip address 172.1.1.1/24 apic1(config-tenant-if)# ipv6 address 2001:1:1::1/64 The scope of the gateway address can be one of the following: • Public—Can be advertised to external Layer 3 networks through routing protocols (BGP, OSPF, EIGRP). • Private—Not advertised to external Layer 3 networks. The optional secondary keyword allows you to configure a secondary gateway address. Examples This example shows the basic configuration of a tenant including assignment to a security domain, creation of a VRF with contracts, and creation of a bridge domain. apic1# configure apic1(config)# tenant exampleCorp apic1(config-tenant)# security domain exampleCorp_dom1 apic1(config-tenant)# vrf context exampleCorp_v1 apic1(config-tenant-vrf)# contract enforce apic1(config-tenant-vrf)# contract provider web apic1(config-tenant-vrf)# contract consumer db apic1(config-tenant-vrf)# contract provider icmp apic1(config-tenant-vrf)# contract consumer icmp apic1(config-tenant-vrf)# exit apic1(config-tenant)# bridge-domain exampleCorp_b1 apic1(config-tenant-bd)# vrf member exampleCorp_v1 apic1(config-tenant-bd)# exit apic1(config-tenant)# interface bridge-domain exampleCorp_b1 apic1(config-tenant-interface)# ip address 172.1.1.1/24 apic1(config-tenant-interface)# ipv6 address 2001:1:1::1/64 apic1(config-tenant-interface)# exit This example shows the VRF configuration specific to a leaf. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# vrf context exampleCorp_v1 tenant exampleCorp apic1(config-leaf-vrf)# ip route 1.2.3.4 5.6.7.8 This example shows the VRF configuration specific to a leaf interface. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# int eth 1/1 apic1(config-leaf-if)# vrf member exampleCorp_v1 tenant exampleCorp Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 35 Configuring Tenants Additional Bridge Domain Configuration What to Do Next Add an application profile, create an application endpoint group (EPG), and associate the EPG to the bridge domain. Additional Bridge Domain Configuration This topic describes the following configurations for a bridge domain: • Configuring a MAC address • Configuring a DHCP relay address • Configuring route leaking for shared services Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic# configure Step 2 tenant tenant-name Enters the tenant configuration mode. Example: apic(config-tenant)# tenant exampleCorp Step 3 interface bridge-domain bd-name Enters tenant interface configuration mode to configure the bridge domain. Example: apic(config-tenant)# interface bridge-domain exampleCorp_bd1 Step 4 mac-address mac-address Example: apic(config-tenant-interface)# mac-address 1234.5678.abcd Step 5 no mac-address Example: apic(config-tenant-interface)# no mac-address Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 36 (Optional) Configures the MAC address to be used in the ARP reply for the pervasive gateway functionality. (Optional) Changes the MAC address to its default. Configuring Tenants Configuring an Enforced Bridge Domain Step 6 Command or Action Purpose [no] ip dhcp relay address tenant tenant-name dhcp-address {application app-name epg epg-name | external-l2 l2-epg-name | external-l3 l3-epg-name} (Optional) Sets or removes a DHCP relay address for the bridge-domain along with any supported options. Example: apic(config-tenant-interface)# ip dhcp relay address 192.0.20.1 tenant exampleCorp application app1 epg epg1 Step 7 [no] {ip | ipv6} shared address address/mask-length provider application app-name epg epg-name Example: apic(config-tenant-interface)# ip shared address 7.8.9.1/24 provider application app2 epg epg2 Step 8 [no] {ip | ipv6} shared address address/mask-length consumer application any epg any (Optional) Route leaking is allowed across VRFs to provide common services like DHCP, DNS for multiple tenant VRFs. Shared service is enabled by marking subnets as provider or consumer subnets and specifying the EPGs providing the shared service. (Optional) See the previous step. Example: apic(config-tenant-interface)# ip shared address 3.2.3.4/24 consumer application any epg any Examples apic1# configure apic1(config)# tenant exampleCorp apic1(config-tenant)# interface bridge-domain exampleCorp_bd1 apic1(config-tenant-interface)# mac-address 1234.5678.abcd apic(config-tenant-interface)# ip dhcp relay address 192.0.20.1 tenant exampleCorp application app1 epg epg1 apic1(config-tenant-interface)# ip shared address 1.2.3.4/24 provider application any apic1(config-tenant-interface)# ip shared address 3.2.3.4/24 consumer application any epg any apic1(config-tenant-interface)# exit apic1(config-tenant)# exit apic1(config)# tenant my_dhcp_provider apic1(config-tenant)# interface bridge-domain bd_dhcp apic1(config-tenant-interface)# ip shared address 7.8.9.1/24 provider application app2 epg epg2 Configuring an Enforced Bridge Domain An enforced bridge domain (BD) configuration entails creating an endpoint in a subject endpoint group (EPG) which can only ping subnet gateways within the associated bridge domain. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 37 Configuring Tenants Configuring an Enforced Bridge Domain Using the Basic GUI With this configuration, you can then create a global exception list of IP addresses which can ping any subnet gateway. Figure 1: Enforced Bridge Domain Note • The exception IP addresses can ping all of the BD gateways across all of your VRFs. • A loopback interface configured for an L3 out does not enforce reachability to the IP address that is configured for the subject loopback interface. • When an eBGP peer IP address exists in a different subnet than the subnet of the L3out interface, the peer subnet must be added to the allowed exception subnets. Otherwise, eBGP traffic is blocked because the source IP address exists in a different subnet than the L3out interface subnet. Configuring an Enforced Bridge Domain Using the Basic GUI Procedure Step 1 Step 2 Log in to the APIC GUI, and on the menu bar, click TENANT > Add Tenant. In the Create Tenant dialog box, perform the following tasks: a) In the Name field, enter a tenant name. b) Click the Security Domains + icon to open the Create Security Domain dialog box. c) In the Name field, enter a security domain name and click Submit. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 38 Configuring Tenants Configuring an Enforced Bridge Domain Using the NX-OS Style CLI d) In the Create Tenant dialog box, check the check box for the security domain that you created, and click Submit. Step 3 In the Navigation pane, expand Tenant-name > Networking, drag the VRF icon to the canvas to open the Create VRF dialog box, and perform the following tasks: a) In the Name field, enter the VRF name. b) Select the BD Enforcement Status check box. c) Click Submit to complete the VRF configuration. To confirm enforced bridge domain configuration, expand Fabric > Fabric Policies > Global policies > Exception List and confirm the presence of the BD Enforced Exception list. Configuring an Enforced Bridge Domain Using the NX-OS Style CLI This section provides information on how to configure your enforced bridge domain using the NX-OS style command line interface (CLI). Procedure Step 1 Create and enable the tenant: Example: In the following example ("cokeVrf") is created and enabled. apic1(config-tenant)# vrf context cokeVrf apic1(config-tenant-vrf)# bd-enforce enable apic1(config-tenant-vrf)# exit apic1(config-tenant)#exit Step 2 Add the subnet to the exception list. Example: apic1(config)#bd-enf-exp-ip add1.2.3.4/24 apic1(config)#exit You can confirm if the enforced bridge domain is operational using the following type of command: apic1# show running-config all | grep bd-enf bd-enforce enable bd-enf-exp-ip add 1.2.3.4/24 The following command removes the subnet from the exception list: apic1(config)# no bd-enf-exp-ip 1.2.3.4/24 apic1(config)#tenant coke apic1(config-tenant)#vrf context cokeVrf What to Do Next To disable the enforced bridge domain run the following command: apic1(config-tenant-vrf)# no bd-enforce enable Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 39 Configuring Tenants Creating an Application Endpoint Group Creating an Application Endpoint Group This topic describes the following steps in the basic provisioning of a static application EPG: 1 Create an application profile within the tenant 2 Create an EPG in the application profile 3 Assign a bridge domain to the EPG 4 Deploy the EPG to a Layer 2 interface Before You Begin Before you can create an application profile and an application endpoint group (EPG), you must create a VLAN domain, tenant, VRF, and bridge domain. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Enters the tenant configuration mode. Example: apic1(config)# tenant exampleCorp Step 3 [no] application app-name Creates an application profile and enters application profile configuration mode. Example: apic1(config-tenant)# application OnlineStore Step 4 [no] epg epg-name Creates (or deletes) an EPG in the application profile and enters EPG configuration mode. Example: apic1(config-tenant-app)# epg exampleCorp_webepg1 Step 5 [no] bridge-domain member epg-name Associates the EPG to the bridge domain. Every EPG must belong to a BD. Example: apic1(config-tenant-app-epg)# bridge-domain member exampleCorp_b1 Step 6 exit Example: apic1(config-tenant-app-epg)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 40 Returns to the tenant application configuration mode. Configuring Tenants Creating an Application Endpoint Group Step 7 Command or Action Purpose exit Returns to the tenant configuration mode. Example: apic1(config-tenant-app)# exit Step 8 Returns to the global configuration mode. exit Example: apic1(config-tenant)# exit Step 9 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 10 interface type Specifies the interface that you are configuring. For an Ethernet port, use “ethernet slot / port.” Example: apic1(config-leaf)# interface eth 1/2 Step 11 apic1(config-leaf-if)# switchport (Optional) Because layer 2 is the default state of a port, this command is only needed when the port must be converted from a layer 3 configuration. vlan-domain member domain-name Associates the interface with a VLAN domain. switchport Example: Step 12 Example: apic1(config-leaf-if)# vlan-domain member dom1 Step 13 switchport trunk allowed vlan vlan-id tenant tenant-name app app-name epg epg-name Example: apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp application OnlineStore epg exampleCorp_webepg1 Deploys the EPG on the interface and identifies the EPG through EPG-to-VLAN mapping. This configuration applies only to static EPG deployment. If the VLAN is in use for another EPG or external SVI, you must delete the VLAN configuration before using it for this EPG. Note The interface must be associated with a VLAN domain or this command is rejected. Examples This example shows how to create an application EPG deployed to a layer 2 port. apic1# configure apic1(config)# tenant exampleCorp apic1(config-tenant)# application OnlineStore apic1(config-tenant-app)# epg exampleCorp_webepg1 apic1(config-tenant-app-epg)# bridge-domain member exampleCorp_b1 apic1(config-tenant-app-epg)# exit apic1(config-tenant-app)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 41 Configuring Tenants Configuring Legacy Forwarding Mode in the Bridge Domain apic1(config-tenant)# exit apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/2 apic1(config-leaf-if)# switchport apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp application OnlineStore epg exampleCorp_webepg1 This example shows how to deploy the EPG to a port channel. apic1(config)# leaf 101 apic1(config-leaf)# interface port-channel po1 apic1(config-leaf-if)# switchport apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp application OnlineStore epg exampleCorp_webepg1 What to Do Next Map a VLAN on a port to the EPG. Configuring Legacy Forwarding Mode in the Bridge Domain Legacy forwarding mode allows switching and routing without the use of contracts or EPGs. In this mode, the VLAN on a port directly maps to a bridge domain. The legacy forwarding vlan command automatically creates all necessary objects so that no EPG-related configuration is required. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: configure Step 2 tenant tenant-name Enters the tenant configuration mode. Example: apic1(config)# tenant exampleCorp Step 3 bridge-domain bd-name Enters tenant interface configuration mode to configure the bridge domain. Example: apic1(config-tenant)# bridge-domain exampleCorp_b1 Step 4 [no] legacy forwarding vlan vlan-id vlan-domain vlan-domain-name Example: apic1(config-tenant-bd)# legacy-forwarding vlan 50 vlan-domain dom1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 42 Maps the VLAN to the bridge domain. Configuring Tenants Configuring Legacy Forwarding Mode in the Bridge Domain Step 5 Command or Action Purpose exit Returns to the tenant configuration mode. Example: apic1(config-tenant-bd)# exit Step 6 Returns to the global configuration mode. exit Example: apic1(config-tenant)# exit Step 7 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 8 interface type Specifies the interface that you are configuring. For an Ethernet port, use ethernet slot/port. Example: apic1(config-leaf)# interface eth 1/1 Step 9 [no] switchport trunk allowed vlan vlan-id tenant tenant-name legacy-forwarding Example: Enables the VLAN on the interface and associates it to the tenant bridge domain that uses the VLAN in the legacy forwarding mode. apic1(config-leaf-if)# switchport trunk allowed vlan 50 tenant exampleCorp legacy-forwarding Examples This example shows how to configure legacy forwarding mode for forwarding between bridge domains. apic1# configure apic1(config)# tenant exampleCorp apic1(config-tenant)# bridge-domain exampleCorp_b1 apic1(config-tenant-bd)# legacy-forwarding vlan 50 vlan-domain dom1 apic1(config-tenant-bd)# exit apic1(config-tenant)# bridge-domain exampleCorp_b2 apic1(config-tenant-bd)# legacy-forwarding vlan 60 vlan-domain dom1 apic1(config-tenant-bd)# exit apic1(config-tenant)# exit apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/1 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk allowed vlan 50 tenant exampleCorp legacy-forwarding apic1(config-leaf-if)# exit apic1(config-leaf)# interface eth 1/2 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk allowed vlan 60 tenant exampleCorp legacy-forwarding Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 43 Configuring Tenants Configuring Contracts Configuring Contracts Contracts are configured under a tenant with the following tasks: • Define filters as access lists • Define the contract and subjects • Link the contract to an EPG The tasks need not follow this order. For example, you can link a contract name to an EPG before you have defined the contract. Note Filters (ACLs) in APIC use match instead of permit | deny as in the traditional NX-OS ACL. The purpose of a filter entry is only to match a given traffic flow. The traffic will be permitted or denied when the ACL is applied on a contract or on a taboo contract. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Creates a tenant if it does not exist and enters the tenant configuration mode. Example: tenant exampleCorp Step 3 access-list acl-name Creates an access list (filter) that can be used in a contract. Example: apic1(config-tenant)# access-list http_acl Step 4 match {arp | icmp | ip} Example: (Optional) Creates a rule to match traffic of the selected protocol. apic1(config-tenant-acl)# match arp Step 5 match {tcp | udp} [src from[-to]] [dest from[-to]] Example: apic1(config-tenant-acl)# match tcp dest 80 apic1(config-tenant-acl)# match tcp dest 443 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 44 (Optional) Creates a rule to match TCP or UDP traffic. Configuring Tenants Configuring Contracts Step 6 Command or Action Purpose match raw options (Optional) Creates a rule to match a raw vzEntry. Example: apic1(config-tenant-acl)# Step 7 Returns to the tenant configuration mode. exit Example: apic1(config-tenant-acl)# exit Step 8 contract contract-name Creates a contract and enters the contract configuration mode. Example: apic1(config-tenant)# contract web80 Step 9 subject subject-name Creates a contract subject and enters the subject configuration mode. Example: apic1(config-tenant-contract)# subject web80 Step 10 [no] access-group acl-name [in | out | both] (Optional) Adds (removes) an access list from the contract, specifying the direction of the traffic to be Example: matched. apic1(config-tenant-contract-subj)# access-group http_acl both Step 11 [no] label name label-name {provider | consumer} (Optional) Adds (removes) a provider or consumer label to the subject. Example: apic1(config-tenant-contract-subj)# Step 12 [no] label match {provider | consumer} [any | one | all | none] Example: apic1(config-tenant-contract-subj)# (Optional) Specifies the match type for the provider or consumer label: • any—Match if any label is found in the contract relation. • one—Match if exactly one label is found in the contract relation. • all—Match if all labels are found in the contract relation. • none—Match if no labels are found in the contract relation. Step 13 Returns to the contract configuration mode. exit Example: apic1(config-tenant-contract-subj)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 45 Configuring Tenants Configuring Contracts Step 14 Command or Action Purpose exit Returns to the tenant configuration mode. Example: apic1(config-tenant-contract)# exit Step 15 application app-name Enters application configuration mode. Example: apic1(config-tenant)# application OnlineStore Step 16 epg epg-name Enters configuration mode for the EPG to be linked to the contract. Example: apic1(config-tenant-app)# epg exampleCorp_webepg1 Step 17 bridge-domain member bd-name Specifies the bridge domain for this EPG. Example: apic1(config-tenant-app-epg)# bridge-domain member exampleCorp_bd1 Step 18 contract provider provider-contract-name Example: apic1(config-tenant-app-epg)# contract provider web80 Step 19 Specifies the provider contract for this EPG. Communication with this EPG can be initiated from other EPGs as long as the communication complies with this provider contract. contract consumer consumer-contract-name Specifies the consumer contract for this EPG. The endpoints in this EPG may initiate communication with any endpoint in an EPG Example: apic1(config-tenant-app-epg)# contract that is providing this contract. consumer rmi99 Examples This example shows how to create and apply contracts to an EPG. apic1# configure apic1(config)# tenant exampleCorp # CREATE FILTERS apic1(config-tenant)# access-list http_acl apic1(config-tenant-acl)# match tcp dest 80 apic1(config-tenant-acl)# match tcp dest 443 apic1(config-tenant-acl)# exit # CREATE CONTRACT WITH FILTERS apic1(config-tenant)# contract web80 apic1(config-tenant-contract)# subject web80 apic1(config-tenant-contract-subj)# access-group http_acl both apic1(config-tenant-contract-subj)# exit apic1(config-tenant-contract)# exit # ASSOCIATE CONTRACTS TO EPG Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 46 Configuring Tenants Contract Inheritance apic1(config-tenant)# application OnlineStore apic1(config-tenant-app)# epg exampleCorp_webepg1 apic1(config-tenant-app-epg)# bridge-domain member exampleCorp_bd1 apic1(config-tenant-app-epg)# contract consumer rmi99 apic1(config-tenant-app-epg)# contract provider web80 apic1(config-tenant-app-epg)# exit apic1(config-tenant-app)#exit apic1(config-tenant)#exit # ASSOCIATE PORT AND VLAN TO EPG apic1(config)#leaf 101 apic1(config-leaf)# interface ethernet 1/4 apic1(config-leaf-if)# switchport trunk allowed vlan 102 tenant exampleCorp application OnlineStore epg exampleCorp_webepg1 This example shows a simpler method for defining a contract by declaring the filters inline in the contract itself. apic1# configure apic1(config)# tenant exampleCorp apic1(config-tenant)# contract web80 apic1(config-tenant-contract)# match tcp 80 apic1(config-tenant-contract)# match tcp 443 Contract Inheritance About Contract Inheritance To streamline associating contracts to new EPGs, you can now enable an EPG to inherit all the (provided and consumed) contracts associated directly to another EPG in the same tenant. Contract inheritance can be configured for application, microsegmented, L2Out, and L3Out EPGs. With Release 3.x, you can also configure contract inheritance for Inter-EPG contracts, both provided and consumed. Inter-EPG contracts are supported on Cisco Nexus 9000 Series switches with EX or FX at the end of their model name or later models. You can enable an EPG to inherit all the contracts associated directly to another EPG, using the APIC GUI, NX-OS style CLI, and the REST API. Figure 2: Contract Inheritance In the diagram above, EPG A is configured to inherit Provided-Contract 1 and 2 and Consumed-Contract 3 from EPG B (contract master for EPG A). Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 47 Configuring Tenants Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI Use the following guidelines when configuring contract inheritance: • Contract inheritance can be configured for application, microsegmented (uSeg), external L2Out EPGs, and external L3Out EPGs. The relationships must be between EPGs of the same type. • Both provided and consumed contracts are inherited from the contract master when the relationship is established. • Contract masters and the EPGs inheriting contracts must be within the same tenant. • Changes to the masters’ contracts are propagated to all the inheritors. If a new contract is added to the master, it is also added to the inheritors. • An EPG can inherit contracts from multiple contract masters. • Contract inheritance is only supported to a single level (cannot be chained) and a contract master cannot inherit contracts. • Contract subject label and EPG label inheritance is supported. • Whether an EPG is directly associated to a contract or inherits a contract, it consumes entries in TCAM. So contract scale guidelines still apply. For more information, see the Verified Scalability Guide for your release. • vzAny security contracts and taboo contracts are not supported. For information about configuring Contract Inheritance and viewing inherited and standalone contracts, see Cisco APIC Basic Configuration Guide. Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI To configure contract inheritance for application or uSeg EPGs, use the following commands: Before You Begin Configure the tenant, application profile, and bridge-domain to be used by the EPGs. Configure the contracts to be shared by the EPGs at the VRF level. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Example: apic1# (config) tenant Tn1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 48 Creates or specifies the tenant to be configured; and enters into tenant configuration mode. Configuring Tenants Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI Step 3 Command or Action Purpose application application-name Creates or specifies an application and enters into application mode. Example: apic1(config-tenant)# application AP1 Step 4 epg epg-name [type micro-segmented] Example: apic1(config-tenant-app)# epg AEPg403 Creates or specifies the application or uSeg EPG to be configured and enters into EPG configuration mode. For uSeg EPGs add the type. In this example, this is the application EPG contract master. Step 5 bridge-domain member bd-name Associates the EPG with the bridge domain. Example: apic1(config-tenant-app-epg)# bridge-domain member T1BD1 Step 6 contract consumer contract-name Adds a contract to be consumed by this EPG. Example: apic1(config-tenant-app-epg)# contract consumer cctr5 Step 7 contract provider [label label] Example: Adds a contract to be provided by this EPG, including an optional list of subject or EPG labels (must be previously configured). apic1(config-tenant-app-epg)# contract provider T1ctrl_cif Step 8 Exits the configuration mode exit Example: apic1(config-tenant-app-epg)# exit Step 9 epg epg-name [type micro-segmented] Example: apic1(config-tenant-app)# epg AEPg404 Creates or specifies the application or uSeg EPG to be configured and enters into EPG configuration mode. For uSeg EPGs add the type. In this example, this is the EPG inheriting contracts. Step 10 bridge-domain member bd-name Associates the EPG with the bridge domain. Example: apic1(config-tenant-app-epg)# bridge-domain member T1BD1 Step 11 inherit-from-epg application application-nameepg EPG-contract-master-name] Configures this EPG to inherit contracts from the EPG contract master. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 49 Configuring Tenants Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI Command or Action Purpose Example: apic1(config-tenant-app-epg)# inherit-from-epg application AP1 epg AEPg403 Step 12 exit Exits the configuration mode Example: apic1(config-tenant-app-epg)# exit Step 13 epg epg-name [type micro-segmented] Example: Step 14 Creates or specifies the application or uSeg EPG to be configured and enters into EPG configuration mode. apic1(config-tenant-app)# epg uSeg1_403_10 type micro-segmented In this example, this is the uSeg EPG contract master. bridge-domain member bd-name Associates the EPG with the bridge domain. Example: apic1(config-tenant-app-epg)# bridge-domain member T1BD1 Step 15 contract provider [label label] Example: Adds a contract to be provided by this EPG, including an optional list of subject or EPG labels (must be previously configured). apic1(config-tenant-app-epg)# contract provider T1ctrl_uSeg_l3out Step 16 attribute-logical-expression logical-expression Adds a logical expression to the uSeg EPG as matching criteria. Example: apic1(config-tenant-app-epg)# attribute-logical-expression 'ip equals 192.168.103.10 force' Step 17 exit Exits the configuration mode Example: apic1(config-tenant-app-epg)# exit Step 18 epg epg-name [type micro-segmented] Example: Step 19 apic1(config-tenant-app)# epg uSeg1_403_30 type micro-segmented In this example, this is the uSeg EPG that inherits contracts from the EPG contract master. bridge-domain member bd-name Associates the EPG with the bridge domain. Example: apic1(config-tenant-app-epg)# bridge-domain member T1BD1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 50 Creates or specifies the application or uSeg EPG to be configured and enters into EPG configuration mode. Configuring Tenants Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI Command or Action Step 20 Purpose attribute-logical-expression logical-expression Adds a logical expression to the uSeg EPG as criteria. Example: apic1(config-tenant-app-epg)# attribute-logical-expression 'ip equals 192.168.103.30 force' Step 21 inherit-from-epg application application-name Configures this EPG to inherit contracts from the EPG contract master. epg EPG-contract-master-name Example: apic1(config-tenant-app-epg)# inherit-from-epg application AP1 epg uSeg1_403_10 Step 22 Exits the configuration mode exit Example: apic1(config-tenant-app-epg)# exit Step 23 Exits the configuration mode exit Example: apic1(config-tenant-app)# exit Step 24 Exits the configuration mode exit Example: apic1(config-tenant)# exit Step 25 exit Exits the configuration mode Example: apic1(config)# exit ifav90-ifc1# show running-config tenant Tn1 application AP1 # Command: show running-config tenant Tn1 application AP1 # Time: Fri Apr 28 17:28:32 2017 tenant Tn1 application AP1 epg AEPg403 bridge-domain member T1BD1 contract consumer cctr5 imported contract provider T1ctr1_cif exit epg AEPg404 bridge-domain member T1BD1 inherit-from-epg application AP1 epg AEPg403 exit epg uSeg1_403_10 type micro-segmented bridge-domain member T1BD1 contract provider T1Ctr1_uSeg_l3out attribute-logical-expression 'ip equals 192.168.103.10 force' exit epg uSeg1_403_30 type micro-segmented bridge-domain member T1BD1 attribute-logical-expression 'ip equals 192.168.103.30 force' inherit-from-epg application AP1 epg uSeg1_403_10 exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 51 Configuring Tenants Configuring L2Out EPG Contract Inheritance Using the NX-OS Style CLI exit exit Configuring L2Out EPG Contract Inheritance Using the NX-OS Style CLI To configure contract inheritance for an external L2Out EPG, use the following commands: Before You Begin Configure the tenant, VRF, and bridge-domain to be used by the EPGs. Configure the Layer 2 outside network (L2Out) that the EPGs will use. Configure the contracts to be shared by the EPGs, at the VRF level. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Example: Creates or specifies the tenant to be configured; and enters into tenant configuration mode. apic1(config)# tenant Tn1 Step 3 external-l2 epg external-l2-epg-name Example: Configures or specifies an external L2Out EPG. In this example, this is the L2out contract master. apic1(config-tenant)# external-l2 epg l2out1:l2Ext1 Step 4 bridge-domain member bd-name Associates the L2Out EPG with a bridge domain. Example: apic1(config-tenant-l2ext-epg)# bridge-domain member T1BD1 Step 5 contract provider contract-name [label label] Adds a contract to be provided by this EPG. Example: apic1(config-tenant-l2ext-epg)# contract provider T1ctr_tcp Step 6 exit Exits the configuration mode Example: apic1(config-tenant-l2ext-epg)# exit Step 7 external-l2 epg external-l2-epg-name Example: apic1(config-tenant)# external-l2 epg L2out12:l2Ext12 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 52 Configures an external L2Out EPG. In this example, this is the EPG that inherits contracts from the L2out contract master. Configuring Tenants Configuring L2Out EPG Contract Inheritance Using the NX-OS Style CLI Step 8 Command or Action Purpose bridge-domain member bd-name Associates the L2out EPG with the bridge domain. Example: apic1(config-tenant-l2ext-epg)# bridge-domain member T1BD1 Step 9 inherit-from-epg L2Out-contract-master-name Configures this EPG to inherit contracts from the L2Out contract master. Example: apic1(config-tenant-l2ext-epg)# inherit-from-epg epg l2out1:l2Ext1 Step 10 Exits the configuration mode exit Example: apic1(config-tenant-l2ext-epg)# exit The steps above are taken from the following example: apic1# show running-config tenant Tn1 external-l2 # Command: show running-config tenant Tn1 external-l2 # Time: Thu May 11 13:10:14 2017 tenant Tn1 external-l2 epg l2out1:l2Ext1 bridge-domain member T1BD1 contract provider T1ctr_tcp exit external-l2 epg l2out10:l2Ext10 bridge-domain member T1BD10 contract provider T1ctr_tcp exit external-l2 epg l2out11:l2Ext11 bridge-domain member T1BD11 contract provider T1ctr_udp exit external-l2 epg l2out12:l2Ext12 bridge-domain member T1BD12 inherit-from-epg epg l2out1:l2Ext1 inherit-from-epg epg l2out10:l2Ext10 inherit-from-epg epg l2out11:l2Ext11 inherit-from-epg epg l2out2:l2Ext2 inherit-from-epg epg l2out3:l2Ext3 inherit-from-epg epg l2out4:l2Ext4 inherit-from-epg epg l2out5:l2Ext5 inherit-from-epg epg l2out6:l2Ext6 inherit-from-epg epg l2out7:l2Ext7 inherit-from-epg epg l2out8:l2Ext8 inherit-from-epg epg l2out9:l2Ext9 exit external-l2 epg l2out2:l2Ext2 bridge-domain member T1BD2 contract provider T1ctr_tcp exit external-l2 epg l2out3:l2Ext3 bridge-domain member T1BD3 contract provider T1ctr_tcp exit external-l2 epg l2out4:l2Ext4 bridge-domain member T1BD4 contract provider T1ctr_tcp exit external-l2 epg l2out5:l2Ext5 bridge-domain member T1BD5 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 53 Configuring Tenants Configuring External L3Out EPG Contract Inheritance Using the NX-OS Style CLI contract provider T1ctr_tcp exit external-l2 epg l2out6:l2Ext6 bridge-domain member T1BD6 contract provider T1ctr_tcp exit external-l2 epg l2out7:l2Ext7 bridge-domain member T1BD7 contract provider T1ctr_tcp exit external-l2 epg l2out8:l2Ext8 bridge-domain member T1BD8 contract provider T1ctr_tcp exit external-l2 epg l2out9:l2Ext9 bridge-domain member T1BD9 contract provider T1ctr_tcp exit exit Configuring External L3Out EPG Contract Inheritance Using the NX-OS Style CLI To configure contract inheritance for an external L3Out EPG, use the following commands: Before You Begin Configure the tenant, VRF, and bridge-domain to be used by the EPGs. Configure the Layer 3 outside network (L3Out) that the EPGs will use. Configure the contracts to be shared by the EPGs, at the VRF level. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Example: Creates or specifies the tenant to be configured; and enters into tenant configuration mode. apic1(config)# tenant Tn1 Step 3 external-l3 epg external-l3-epg-name l3out l3out-name Configures an external L3Out EPG. In this example, this is the L3out contract master. Example: apic1(config-tenant-app)# external-l3 epg l3Ext108 l3out T1L3out1 Step 4 vrf member vrf-name Example: apic1(tenant-l3out)# vrf member T1ctx1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 54 Associates the L3out with the VRF. Configuring Tenants Configuring External L3Out EPG Contract Inheritance Using the NX-OS Style CLI Step 5 Command or Action Purpose match ip ip-address-and-mask Adds a subnet that identifies hosts as part of the EPG and adds the optional shared scope for the subnet. Example: apic1(config-tenant-l3ext-epg)# match ip 192.168.110.0/24 shared Step 6 contract provider contract-name [label label] Adds a contract to be provided by this EPG. Example: apic1(config-tenant-l3ext-epg)# contract provider T1ctrl-L3out Step 7 Exits the configuration mode exit Example: apic1(config-tenant-l3ext-epg)# exit Step 8 external-l3 epg external-l3-epg-name l3out l3out-name Configures an external L3Out EPG. In this example, this is the EPG that inherits contracts from the L3out contract master. Example: apic1(config-tenant-app)# external-l3 epg l3Ext110 l3out T1L3out1 Step 9 vrf member vrf-name Associates the L3out with the VRF. Example: apic1(tenant-l3out)# vrf member T1ctx1 Step 10 match ip ip-address-and-mask Example: Adds a subnet that identifies hosts as part of the EPG and adds the optional shared scope for the subnet. apic1(config-tenant-l3ext-epg)# match ip 192.168.112.0/24 shared Step 11 inherit-from-epg L3Out-contract-master-name Configures this EPG to inherit contracts from the L3Out contract master. Example: apic1(config-tenant-l3ext-epg)# inherit-from-epg l3Ext108 Step 12 Exits the configuration mode exit Example: apic1(config-tenant-l3ext-epg)# exit ifav90-ifc1# show running-config tenant Tn1 external-l3 epg l3Ext110 # Command: show running-config tenant Tn1 external-l3 epg l3Ext110 # Time: Fri Apr 28 17:36:15 2017 tenant Tn1 external-l3 epg l3Ext108 l3out T1L3out1 vrf member T1ctx1 match ip 192.168.110.0/24 shared contract provider T1ctrl-L3out exit external-l3 epg l3Ext110 l3out T1L3out1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 55 Configuring Tenants Configuring Contract Preferred Groups vrf member T1ctx1 match ip 192.168.112.0/24 shared inherit-from-epg epg l3Ext108 exit exit Configuring Contract Preferred Groups About Contract Preferred Groups There are two types of policy enforcements available for EPGs in a VRF with a contract preferred group configured: • Included EPGs: EPGs can freely communicate with each other without contracts, if they have membership in a contract preferred group. This is based on the source-any-destination-any-permit default rule. • Excluded EPGs: EPGs that are not members of preferred groups require contracts to communicate with each other. Otherwise, the default source-any-destination-any-deny rule applies. The contract preferred group feature enables greater control of communication between EPGs in a VRF. If most of the EPGs in the VRF should have open communication, but a few should only have limited communication with the other EPGs, you can configure a combination of a contract preferred group and contracts with filters to control inter-EPG communication precisely. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 56 Configuring Tenants About Contract Preferred Groups EPGs that are excluded from the preferred group can only communicate with other EPGs if there is a contract in place to override the source-any-destination-any-deny default rule. Figure 3: Contract Preferred Group Overview Limitations The following limitations apply to contract preferred groups: • In topologies where an L3Out and application EPG are configured in a Contract Preferred Group, and the EPG is deployed only on a VPC, you may find that only one leaf switch in the VPC has the prefix entry for the L3Out. In this situation, the other leaf switch in the VPC does not have the entry, and therefore drops the traffic. To workaround this issue, you can do one of the following: ◦Disable and reenable the contract group in the VRF ◦Delete and recreate the prefix entries for the L3Out EPG • Also, where the provider or consumer EPG in a service graph contract is included in a contract group, the shadow EPG can not be excluded from the contract group. The shadow EPG will be permitted in the contract group, but it does not trigger contract group policy deployment on the node where the shadow EPG is deployed. To download the contract group policy to the node, you deploy a dummy EPG within the contract group . Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 57 Configuring Tenants Configuring Contract Preferred Groups Using the NX-OS Style CLI Configuring Contract Preferred Groups Using the NX-OS Style CLI You can use the APIC NX-OS style CLI to configure a contract preferred group. In this example, a contract preferred group is configured for a VRF. One of the EPGs using the VRF is included in the preferred group. Before You Begin Create the tenants, VRFs, and EPGs that will consume the contract preferred group. Procedure Step 1 Command or Action Purpose configure Enters configuration mode Example: apic1# configure apic1(config)# Step 2 tenanttenant-name Creates a tenant or enters into tenant configuration mode Example: apic1(config)# tenant tenant64 Step 3 vrf context vrf-name Creates a VRF or enters into VRF configuration mode. Example: apic1(config-tenant)# vrf context vrf64 Step 4 whitelist-blacklist-mix Example: Enables a contract preferred group for the VRF and then returns to tenant configuration mode. apic1(config-tenant-vrf)# whitelist-blacklist-mix apic1(config-tenant-vrf)# exit Step 5 bridge-domain bd-name Creates a bridge-domain for the VRF or enters into BD configuration mode. Example: apic1(config-tenant)# bridge-domain bd64 Step 6 vrf member vrf-name Associates the VRF with the bridge-domain and returns to teanant configuration mode. Example: apic1(config-tenant-bd)# vrf member vrf64 apic1(config-tenant-bd)# exit Step 7 application app-name Creates an application or enters into application configuration mode. Example: apic1(config-tenant)# application app-ldap Step 8 epg epg-name Example: apic1(config-tenant-app)# epg epg-ldap Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 58 Creates an EPG or enters into EPG tenant-app EPG configuration mode. Configuring Tenants Exporting a Contract to Another Tenant Step 9 Command or Action Purpose bridge-domain member bd-name Associates the EPG with the bridge-domain . Example: apic1(config-tenant-app-epg)# bridge-domain member bd64 Step 10 Configures this EPG to be included in the contract preferred group. vrf-blacklist-mode Example: apic1(config-tenant-app-epg)# vrf-blacklist-mode The following example creates a contract preferred group for vrf64 and includes epg-ldap in it. apic1# configure apic1(config)# tenant tenant64 apic1(config-tenant)# vrf context vrf64 apic1(config-tenant-vrf)# whitelist-blacklist-mix apic1(config-tenant-vrf)# exit apic1(config-tenant)# bridge-domain bd64 apic1(config-tenant-bd)# vrf member vrf64 apic1(config-tenant-bd)# exit apic1(config-tenant)# application app-ldap apic1(config-tenant-app)# epg epg-ldap apic1(config-tenant-app-epg)# bridge-domain member bd64 apic1(config-tenant-app-epg)# vrf-blacklist-mode Exporting a Contract to Another Tenant You can export a contract from one tenant and import it to another. In the tenant that imports the contract, the contract can be applied only as a consumer contract. The contract can be renamed during the export. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Enters the tenant configuration mode for the exporting tenant. Example: apic1(config)# tenant RedCorp Step 3 contract contract-name Enters the contract configuration mode for the contract to be exported. Example: apic1(config-tenant)# contract web80 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 59 Configuring Tenants Exporting a Contract to Another Tenant Command or Action Step 4 Purpose scope {application | exportable | tenant | Configures how the contract can be shared. The scope can be: vrf} Example: apic1(config-tenant-contract)# scope exportable • application—Can be shared among the EPGs of the same application. • exportable—Can be shared across tenants. • tenant—Can be shared among the EPGs of the same tenant. • vrf—Can be shared among the EPGs of the same VRF. Step 5 export to tenant other-tenant-name as new-contract-name Exports the contract to the other tenant. You can use the same contract name or you can rename it. Example: apic1(config-tenant-contract)# export to tenant BlueCorp as webContract1 Step 6 exit Returns to the tenant configuration mode. Example: apic1(config-tenant-contract)# exit Step 7 exit Returns to the global configuration mode. Example: apic1(config-tenant)# exit Step 8 tenant tenant-name Enters the tenant configuration mode for the importing tenant. Example: tenant BlueCorp Step 9 application app-name Enters application configuration mode. Example: apic1(config-tenant)# application BlueStore Step 10 epg epg-name Enters configuration mode for the EPG to be linked to the contract. Example: apic1(config-tenant-app)# epg BlueWeb Step 11 contract consumer consumer-contract-name imported Example: apic1(config-tenant-app-epg)# contract consumer webContract1 imported Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 60 Specifies the imported consumer contract for this EPG. The endpoints in this EPG may initiate communication with any endpoint in an EPG that is providing this contract. Configuring Tenants Creating Quota Management Examples This example shows how to export a contract from the tenant RedCorp to the tenant BlueCorp, where it will be a consumer contract. apic# configure apic1(config)# tenant RedCorp apic1(config-tenant)# contract web80 apic1(config-tenant-contract)# scope exportable apic1(config-tenant-contract)# export to tenant BlueCorp as webContract1 apic1(config-tenant-contract)# exit apic1(config-tenant)# exit apic1(config)# tenant BlueCorp apic1(config-tenant)# application BlueStore apic1(config-tenant-application)# epg BlueWeb apic1(config-tenant-application-epg)# contract consumer webContract1 imported Creating Quota Management About APIC Quota Management Configuration Starting in the Cisco Application Policy Infrastructure Controller (APIC) Release 2.3(1), there are limits on number of objects a tenant admin can configure. This enables the admin to limit what managed objects that can be added under a given tenant or globally across tenants. This feature is useful when you want to limit any tenant or group of tenants from exceeding ACI maximums per leaf or per fabric or unfairly consuming a majority of available resources, potentially affecting other tenants on the same fabric. Creating a Quota Management Configuration Using the NX-OS Style CLI This procedure explains how to create a quota management configuration using the NX-OS Style CLI. Procedure Create a quota management configuration using the NX-OS CLI: Example: apic1# conf t apic1(config)# quota fvBD max 100 scope uni/tn-green exceed-action fault apic1(config)# quota fvBD max 1000 scope uni exceed-action fail apic1(config)# quota fvBD max 34 tenant red Syntax: [no] quota <className> max <maxValue> [exceed-action {fail|fault}] \ [{scope <containerDn>| tenant <tenantName> \ [{bridge-domain <bd>|application <ap> [epg <epgName>]}]}] Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 61 Configuring Tenants Creating a Quota Management Configuration Using the NX-OS Style CLI where <className> is the managed object className such as fvBD or fvCtx etc. All the eligible classes accordingly to the presence of the quota flag in the model are accepted. where <maxValue> is the value after which the <exceed-action> is applied. where <exceed-action> is the action to be taken after the <maxValue> is exceeded, can either be: • fail: when you want to fail the transaction exceeding the limit. • fault: raise a fault. where <containerDn> is the tree under which the limit will be enforced. "uni" will be across the whole ACI policy model, "tenant green" will be for the tenant green. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 62 CHAPTER 5 Configuring Layer 2 External Connectivity • Configuring Layer 2 External Connectivity, page 63 • Configuring VLAN Domains, page 66 • Configuring Q-in-Q Encapsulation Mapping for EPGs, page 74 • Support Fibre Channel over Ethernet Traffic on the ACI Fabric, page 76 • Configuring 802.1Q Tunnels, page 89 • Configuring Dynamic Breakout Ports, page 94 • Microsegmentation on Virtual Switches, page 98 • Configuring Microsegmentation on Bare-Metal , page 100 • Configuring Layer 2 IGMP Snoop Multicast, page 102 • Configuring Port Security, page 109 • Configuring Proxy ARP, page 116 Configuring Layer 2 External Connectivity Layer 2 External Connectivity represents the switching network between the ACI leaf switches (aka border leaf) and an External Router. The VLAN representing the external L2 network is mapped to one of the bridge-domains within the fabric, which provides the Layer 2 extension for the bridge-domain and lets the EPGs using the bridge-domain talk to the outside network. The outside network is mapped to an EPG, which helps in realizing contracts between different internal applications and different L2 outside VLANs across nodes. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 63 Configuring Layer 2 External Connectivity Configuring Layer 2 External Connectivity Caution Do not mix the Advanced GUI and the CLI, when doing per-interface configuration on APIC. Configurations performed in the GUI, may only partially work in the NX-OS CLI. For example, if you configure a switch port in the GUI at Tenants > tenant-name > Application Profiles > application-profile-name > Application EPGs > EPG-name > Static Ports > Deploy Static EPG on PC, VPC, or Interface Then you use the show running-config command in the NX-OS style CLI, you receive output such as: leaf 102 interface ethernet 1/15 switchport trunk allowed vlan 201 tenant t1 application ap1 epg ep1 exit exit If you use these commands to configure a static port in the NX-OS style CLI, the following error occurs: apic1(config)# leaf 102 apic1(config-leaf)# interface ethernet 1/15 apic1(config-leaf-if)# switchport trunk allowed vlan 201 tenant t1 application ap1 epg ep1 No vlan-domain associated to node 102 interface ethernet1/15 encap vlan-201 This occurs because the CLI has validations that are not performed by the APIC GUI. For the commands from the show running-config command to function in the NX-OS CLI, a vlan-domain must have been previously configured. The order of configuration is not enforced in the GUI. The configuration for Layer2 external connectivity is similar to a static application EPG, where you map a VLAN on a node port to an EPG and map the EPG to a bridge-domain to provide/consume contracts. Procedure Command or Action Step 1 Purpose Access configuration mode. Example: apic1# configure Step 2 Enter tenant configuration mode. Example: apic1(config)# tenant exampleCorp Step 3 [no] external-l2 epg epg-name Example: apic1(config-tenant)# external-l2 epg extendBD1 Step 4 Assign a bridge domain to the EPG. Example: apic1(config-tenant-extl2epg)# bridge-domain member bd1 Step 5 Return to tenant configuration mode. Example: apic1(config-tenant-extl2epg)# exit Step 6 Return to global configuration mode. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 64 Create (or delete ) an external layer 2 EPG. Configuring Layer 2 External Connectivity Configuring Layer 2 External Connectivity Command or Action Purpose Example: apic1(config-tenant)# exit Step 7 Specify the leaf to be configured. Example: apic1(config)# leaf 101 Step 8 Specify a port for the external EPG. Example: apic1(config-leaf)# interface eth 1/2 Step 9 By default, a port is in Layer 2 trunk mode. If the port is in Layer 3 mode, convert it to Layer 2 trunk mode using this command. Example: apic1(config-leaf-if)# switchport Step 10 Associate the interface with a VLAN domain. Example: apic1(config-leaf-if)# vlan-domain member dom1 Step 11 Assigns a VLAN on the leaf port and maps the VLAN to a layer Note 2 external EPG, with the switchport trunk allowed vlan vlan-id tenant tenant-name external-l2 epg epg-name command. Example: apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp external-l2 epg extendBD1 Step 12 Assign a VLAN on the leaf port and map the VLAN to an external Note SVI with the switchport {trunk allowed | trunk native | access} vlan vlan-id tenant tenant-name external-svi command. Example: apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp external-svi The interface must be associated with a VLAN domain or this command is rejected. The interface must be associated with a VLAN domain or this command is rejected. Examples This example shows how to deploy a layer 2 port for external connectivity. apic1# configure apic1(config)# tenant exampleCorp apic1(config-tenant)# external-l2 epg extendBD1 apic1(config-tenant-extl2epg)# bridge-domain member bd1 apic1(config-tenant-extl2epg)# exit apic1(config-tenant)# exit apic1(config)# leaf 101 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 65 Configuring Layer 2 External Connectivity Configuring VLAN Domains apic1(config-leaf)# interface eth 1/2 apic1(config-leaf-if)# switchport apic1(config-leaf-if)# switchport mode trunk apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp external-l2 epg extendBD1 This example shows how to deploy a layer 2 port channel or vPC for external connectivity. ... apic1(config)# leaf 101 apic1(config-leaf)# interface port-channel po1 apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp external-l2 epg extendBD1 These examples show how to configure SVI on a layer 2 interface for external connectivity. apic1(config)# leaf 101 pic1(config-leaf)# interface ethernet 1/5 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp external-svi apic1(config-leaf-if)# no switchport trunk allowed vlan 10 tenant exampleCorp external-svi apic1(config-leaf)# interface ethernet 1/37 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport access vlan 11 tenant exampleCorp external-svi apic1(config-leaf-if)# no switchport access vlan 11 tenant exampleCorp external-svi apic1(config-leaf)# interface port-channel po34 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk native vlan 12 tenant exampleCorp external-svi apic1(config-leaf-if)# no switchport trunk native vlan 12 tenant exampleCorp external-svi Configuring VLAN Domains About VLAN Domains ACI fabric can be partitioned into groups of 4K VLANs to allow a large number of layer 2 domains across the fabric, which can be used by multiple tenants. A VLAN domain represents a set of VLANs that can be configured on group of nodes and ports. VLAN domains let multiple tenants share and independently manage common fabric resources such as nodes, ports, and VLANs. A tenant can be provided access to one or more VLAN domains. For more information about VLAN pools, see Endpoint Groups in the ACI Policy Model chapter in Cisco Application Centric Infrastructure Fundamentals. VLAN domains can be static or dynamic. Static VLAN domains support static VLAN pools, while dynamic VLAN domains can support both static and dynamic VLAN pools. VLANs in static pools are managed by the user and are used for applications such as connectivity to bare metal hosts. VLANs in the dynamic pool are allocated and managed by the APIC without user intervention and are used for applications such as VMM. The default type for VLAN domains and VLAN pools within the domain is static. The fabric administrator performs the following steps before tenants can start using the fabric resources for their L2/L3 configurations: 1 Create VLAN domains and assign VLANs in each VLAN domain. 2 Assign the external facing ports on the leaf switches to one or more VLAN domains. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 66 Configuring Layer 2 External Connectivity Basic VLAN Domain Configuration 3 Convert a port to L2/L3 by using the [no] switchport command. The default state of a port is L2(switchport) in trunk mode. 4 For an L2 port, set the scope of a VLAN on a port to be global or local. The default is global. The fabric administrator can update any configuration in these steps even after VLAN domains are assigned to tenants and are in use by tenant applications. A Note About Spanning Tree and VLAN Domains Although the ACI fabric does not participate in spanning tree, it can partition a spanning tree domain based on access policy configuration. ACI does not rely on a bridge domain or its settings to determine spanning tree domains. Instead, leaf switches flood BPDUs within the same VLAN encapsulation, if a VLAN Pool is assigned to EPG domains. The VLAN pool assigned to EPG domains ultimately serves as the spanning tree domain. Using multiple EPG domains tied to different VLAN Pools does not allow BPDUs to flood across endpoints properly, even if they are all using the same VLAN ID. The type of EPG domain, (physical or Layer 2 external), does not change this behavior. Because the ACI Fabric floods all BPDUs from all devices within a spanning-tree domain, this may trigger behaviors on external devices that are verifying BPDU info, such as the MAC address per interface. An example of a feature that activates is "spanning-tree EtherChannel misconfig guard" found on IOS devices. These features should be taken into account when utilizing ACI as a Layer 2 Tunnel. Note Multiple Spanning Tree (MST) is not supported on interfaces configured with the Per Port VLAN feature (configuring multiple EPGs on a leaf switch using the same VLAN ID with localPort scope). Basic VLAN Domain Configuration Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] vlan-domain domain-name Creates a VLAN domain or edits an existing domain. Include the dynamic keyword to create a dynamic VLAN pool. The [dynamic] default is static. Example: apic1(config)# vlan-domain dom2 dynamic Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 67 Configuring Layer 2 External Connectivity Advanced VLAN Domain Configuration Step 3 Command or Action Purpose [no] vlan range [dynamic] Assigns a range or a comma-separated list of VLANs to the VLAN domain. Example: A VLAN can be either static or dynamic. A static VLAN is configured by the user, such as for providing connectivity from a host to an external switched network, while VLANs in the dynamic range are configured internally by an APIC application, such as a VMM or L4-L7 services. The default type is static. apic1(config-vlan)# vlan 1000-1999,4001 Note A static domain cannot contain dynamic VLANs. A VLAN on a given port can map to only one vlan-domain. This is enforced during configuration. Examples This example shows how to configure basic VLAN domains. apic1# configure apic1(config)# vlan-domain dom1 apic1(config-vlan)# vlan 1000-1999,4001 apic1(config-vlan)# exit apic1(config)# vlan-domain dom2 dynamic apic1(config-vlan)# vlan 101-200 apic1(config-vlan)# vlan 301-400 dynamic Advanced VLAN Domain Configuration Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] vlan-domain domain-name [dynamic] [type {phys | l2ext | l3ext}] Creates a VLAN domain or edits an existing domain. Include the dynamic keyword to create a dynamic VLAN pool. The default is static. Example: The type option is visible and mandatory if one or more of the following conditions exist: apic1(config)# vlan-domain dom1 type phys • If all three vlan-domain types are not present for this domain name • If the three vlan-domain types have different VLAN pools Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 68 Configuring Layer 2 External Connectivity Advanced VLAN Domain Configuration Command or Action Purpose • If the three vlan-domain types share the same VLAN pool but if the pool name differs from the vlan-domain name Step 3 [no] vlan-pool vlan-pool-name Example: apic1(config-leaf)# vlan-pool myVlanPool3 Step 4 Creates a VLAN pool. This command is available only when the type option is present in the vlan-domain command. You must declare the VLAN pool before adding VLANs with the vlan command. [no] vlan range [dynamic] Assigns a range or a comma-separated list of VLANs to the VLAN domain. Example: A VLAN can be either static or dynamic. A static VLAN is configured by the user, such as for providing connectivity from a host to an external switched network, while VLANs in the dynamic range are configured internally by an APIC application, such as a VMM or L4-L7 services. The default type is static. apic1(config-vlan-domain)# vlan 1000-1999,4001 Note A static domain cannot contain dynamic VLANs. A VLAN on a given port can map to only one vlan-domain. This is enforced during configuration. Step 5 show vlan-domain [name Displays vlan-domain usage for applications such as App-EPG, domain-name] [vlan vlan-id] [leaf sub-interface, external SVI, and external-L2. leaf-id] Example: apic1(config-vlan-domain)# show vlan-domain name dom1 vlan 1002 leaf 102 Examples This example shows how to configure a VLAN domain with a VLAN pool. apic1# configure (config)# vlan-domain dom1 type phys (config-vlan-domain)# vlan-pool myVlanPool3 (config-vlan-domain)# vlan 1000-1999, 4001 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 69 Configuring Layer 2 External Connectivity Associating a VLAN Domain to a Port Associating a VLAN Domain to a Port Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 leaf node-id1-node-id2 Specifies the pair of leafs to be configured. Example: apic1(config)# leaf 101-102 Step 3 interface type Specifies a port or range of ports to be associated with the VLAN domain. Example: apic1(config-leaf)# int eth 1/1-24 Step 4 [no] vlan-domain member domain-name Assigns the specified ports to the VLAN domain. Example: apic1(config-leaf-if)# vlan-domain member dom1 Step 5 [no] switchport Example: By default, a port is in Layer 2 trunk mode. If the port is in Layer 3 mode, it must be converted to Layer 2 trunk mode using this command. apic1(config-leaf-if)# switchport Step 6 [no] switchport vlan scope local Example: apic1(config-leaf-if)# switchport vlan scope local (Optional) By default, the scope of a VLAN is global to the node. One VLAN can be mapped to only one EPG in the node. When the VLAN scope is local to the port, the mapping from VLAN to EPG can be different for different ports on the same node. To return the scope to global, use the no command prefix. Step 7 show vlan-domain [name domain-name] Displays vlan-domain usage for applications such as App-EPG, external SVI, and external-L2. [vlan vlan-id] [leaf leaf-id] Example: apic1(config-leaf-if)# show vlan-domain name dom1 vlan 1002 leaf 102 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 70 Configuring Layer 2 External Connectivity Associating a VLAN Domain to a Port-Channel Examples This example shows how to associate a VLAN domain to ports. apic1# configure (config) # leaf 101-102 (config-leaf) # int eth 1/1-24 (config-leaf-if) # vlan-domain member dom1 (config-leaf) # int eth 1/1-12 (config-leaf-if) # no switchport (config-leaf) # int eth 1/13-24 (config-leaf-if) # switchport (config) # leaf 101-102 (config-leaf) # int eth 1/1-12 (config-leaf-if) # switchport vlan scope local (config-leaf) # int eth 1/13 (config-leaf-if) # no switchport vlan scope local Associating a VLAN Domain to a Port-Channel Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 leaf node-id1-node-id2 Specifies the pair of leafs to be configured. Example: apic1(config)# leaf 101-102 Step 3 interface port-channel port-channel-name Specifies a port-channel to be associated with the VLAN domain. Example: apic1(config-leaf)# int port-channel pc1 Step 4 [no] vlan-domain member domain-name Assigns the specified port-channel to the VLAN domain. Example: apic1(config-leaf-if)# vlan-domain member dom1 Examples apic1# configure apic1(config)# leaf 101-102 apic1(config-leaf)# int port-channel pc1 apic1(config-leaf-if)# vlan-domain member dom1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 71 Configuring Layer 2 External Connectivity Associating a VLAN Domain to a Template Policy-Group Associating a VLAN Domain to a Template Policy-Group Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 template policy-group policy-group-name Specifies the template policy-group to be configured. Example: apic1(config)# template policy-group myPolGp5 Step 3 [no] vlan-domain member domain-name Assigns the specified template policy-group to the VLAN domain. Example: apic1(config-pol-grp-if)# vlan-domain member dom1 Examples apic1# configure apic1(config)# template policy-group myPolGp5 apic1(config-pol-grp-if)# vlan-domain member dom1 Associating a VLAN Domain to a Template Port-Channel Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 template port-channel policy-group-name Example: apic1(config)# template port-channel myPC7 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 72 Specifies the template port-channel to be configured. Configuring Layer 2 External Connectivity Associating a VLAN Domain to a Virtual Port-Channel Step 3 Command or Action Purpose [no] vlan-domain member domain-name Assigns the specified template port-channel to the VLAN domain. Example: apic1(config-if)# vlan-domain member dom1 Examples apic1# configure apic1(config)# template port-channel myPC7 apic1(config-po-ch-if)# vlan-domain member dom1 Associating a VLAN Domain to a Virtual Port-Channel Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 vpc context leaf node-id1 node-id2 [fex fex-id1 fex-id2] Specifies the VPC and leafs to be configured. Example: apic1(config)# vpc context leaf 101 102 Step 3 interface vpc vpc-name [fex fex-id1 fex-id2] Specifies a port-channel to be associated with the VLAN domain. Example: apic1(config-vpc)# int vpc vpc1 Step 4 [no] vlan-domain member domain-name Assigns the specified VPC to the VLAN domain. Example: apic1(config-vpc-if)# vlan-domain member dom1 Examples apic1# configure apic1(config)# vpc context leaf 101 102 apic1(config-vpc)# int vpc vpc1 apic1(config-vpc-if)# vlan-domain member dom1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 73 Configuring Layer 2 External Connectivity Configuring Q-in-Q Encapsulation Mapping for EPGs Configuring Q-in-Q Encapsulation Mapping for EPGs Q-in-Q Encapsulation Mapping for EPGs Using Cisco APIC, you can map double-tagged VLAN traffic ingressing on a regular interface, PC, or VPC to an EPG. When this feature is enabled, when double-tagged traffic enters the network for an EPG, both tags are processed individually in the fabric and restored to double-tags when egressing the ACI switch. Ingressing single-tagged and untagged traffic is dropped. This feature is only supported on Nexus 9300-FX platform switches. Both the outer and inner tag must be of EtherType 0x8100. MAC learning and routing are based on the EPG port, sclass, and VRF, not on the access encapsulations. QoS priority settings are supported, derived from the outer tag on ingress, and rewritten to both tags on egress. EPGs can simultaneously be associated with other interfaces on a leaf switch, that are configured for single-tagged VLANs. Service graphs are supported for provider and consumer EPGs that are mapped to Q-in-Q encapsulated interfaces. You can insert service graphs, as long as the ingress and egress traffic on the service nodes is in single-tagged encapsulated frames. The following features and options are not supported with this feature: • Per-Port VLAN feature • FEX connections • Mixed Mode is not supported. For example, an interface in Q-in-Q encapsulation mode can have a static path binding to an EPG with double-tagged encapsulation only, not with regular VLAN encapsulation. • STP and the “Flood in Encapsulation” option • Untagged and 802.1p mode • Multi-pod and Multi-Site • Legacy bridge domain • L2Out and L3Out connections • VMM integration • Changing a port mode from routed to Q-in-Q encapsulation mode is not supported • Per-vlan MCP is not supported between ports in Q-in-Q encapsulation mode and ports in regular trunk mode. • When VPC ports are enabled for Q-in-Q encapsulation mode, VLAN consistency checks are not performed. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 74 Configuring Layer 2 External Connectivity Mapping EPGs to Q-in-Q Encapsulated Leaf Interfaces Using the NX-OS Style CLI Mapping EPGs to Q-in-Q Encapsulated Leaf Interfaces Using the NX-OS Style CLI Enable an interface for Q-in-Q encapsulation and associate the interface with an EPG. Before You Begin Create the tenant, application profile, and application EPG that will be mapped with an interface configured for Q-in-Q mode. Procedure Step 1 Command or Action Purpose Configure Enters global configuration mode. Example: apic1# configure Step 2 leaf number Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 interface ethernetslot/port Specifies the interface to be configured. Example: apic1 (config-leaf)# interface ethernet 1/25 Step 4 switchport mode dot1q-tunnel doubleQtagPort Enables an interface for Q-in-Q encapsulation. Example: apic1(config-leaf-if)# switchport mode dot1q-tunnel doubleQtagPort Step 5 switchport trunkqinq outer-vlanvlan-number inner-vlan Associates the interface with an EPG. vlan-number tenant tenant-name application application-name epg epg-name Example: apic1(config-leaf-if)# switchport trunk qinq outer-vlan 202 inner-vlan 203 tenant tenant64 application AP64 epg EPG64 The following example enables Q-in-Q encapsulation (with outer-VLAN ID 202 and inner-VLAN ID 203) on the leaf interface 101/1/25, and associates the interface with EPG64. apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/25 apic1(config-leaf-if)#switchport mode dot1q-tunnel doubleQtagPort apic1(config-leaf-if)# switchport trunk qinq outer-vlan 202 inner-vlan 203 tenant tenant64 application AP64 epg EPG64 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 75 Configuring Layer 2 External Connectivity Support Fibre Channel over Ethernet Traffic on the ACI Fabric Support Fibre Channel over Ethernet Traffic on the ACI Fabric Supporting Fibre Channel over Ethernet Traffic on the ACI Fabric ACI enables you to configure and manage support for Fibre Channel over Ethernet (FCoE) traffic on the ACI fabric. FCoE is a protocol that encapsulates Fibre Channel (FC) packets within Ethernet packets, thus enabling storage traffic to move seamlessly from a Fibre Channel SAN to an Ethernet network. A typical implementation of FCoE protocol support on the ACI fabric enables hosts located on the Ethernet-based ACI fabric to communicate with SAN storage devices located on an FC network. The hosts are connecting through virtual F ports deployed on an ACI leaf switch. The SAN storage devices and FC network are connected through an FCF bridge to the ACI fabric through a virtual NP port, deployed on the same ACI leaf switch as is the virtual F port. Virtual NP ports and virtual F ports are also referred to generically as virtual Fibre Channel (vFC) ports. Note As of release version 2.0(1), FCoE support is limited to 9300-EX hardware. With release version 2.2(x), the N9K-CP3180LC-EX 40 Gigabit Ethernet (GE) ports can be used as F or NP ports. However, If they are enabled for FCoE, they cannot be enabled for 40GE port breakout. Breakout is not supported with FCoE. As of release version 2.2(x), FCoE is also supported on the following FEX Nexus devices: • N2K-C2348UPQ-10GE • N2K-C2348TQ-10GE • N2K-C2248PQ-10GE • B22 FEX for Vendor Blade Servers Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 76 Configuring Layer 2 External Connectivity Supporting Fibre Channel over Ethernet Traffic on the ACI Fabric Topology Supporting FCoE Traffic Through ACI The topology of a typical configuration supporting FCoE traffic over the ACI fabric consists of the following components: Figure 4: ACI Topology Supporting FCoE Traffic • One or more ACI leaf switches configured through FC SAN policies to function as an NPV backbone • Selected interfaces on the NPV-configured leaf switches configured to function as F ports. F ports accommodate FCoE traffic to and from hosts running SAN management or SAN-consuming applications. • Selected interfaces on the NPV-configured leaf switches to function as NP ports. NP ports accommodate FCoE traffic to and from an FCF bridge. The FCF bridge receives FC traffic from fibre channel links typically connecting SAN storage devices and encapsulates the FC packets into FCoE frames for transmission over the ACI fabric to the SAN management Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 77 Configuring Layer 2 External Connectivity FCoE NX-OS Style CLI Configuration or SAN Data-consuming hosts. It receives FCoE traffic and repackages it back to FC for transmission over the fibre channel network. Note In the above ACI topology, FCoE traffic support requires direct connections between the hosts and F ports and direct connections between the FCF device and NP port. APIC servers enable an operator to configure and monitor the FCoE traffic through the APIC Basic GUI, the APIC Advanced GUI, the APIC NX-OS style CLI, or through application calls to the APIC REST API. Topology Supporting FCoE Initialization In order for FCoE traffic flow to take place as described, you'll also need to set up separate VLAN connectivity over which SAN Hosts broadcast FCoE Initialization protocol (FIP) packets to discover the interfaces enabled as F ports. vFC Interface Configuration Rules Whether you set up the vFC network and EPG deployment through the APIC Basic or Advanced GUI, NX-OS style CLI, or the REST API, the following general rules apply across platforms: • F port mode is the default mode for vFC ports. NP port mode must be specifically configured in the Interface policies. • The load balancing default mode is for leaf-switch or interface level vFC configuration is src-dst-ox-id. • One VSAN assignment per bridge domain is supported. • The allocation mode for VSAN pools and VLAN pools must always be static. • vFC ports require association with a VSAN domain (also called Fibre Channel domain) that contains VSANS mapped to VLANS. FCoE NX-OS Style CLI Configuration Configuring FCoE Connectivity Without Policies or Profiles Using the NX-OS Style CLI The following sample NX-OS style CLI sequences configure FCoE connectivity for EPG e1 under tenant t1 without configuring or applying switch-level and interface-level policies and profiles. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 78 Configuring Layer 2 External Connectivity Configuring FCoE Connectivity Without Policies or Profiles Using the NX-OS Style CLI Procedure Command or Action Purpose Step 1 Under the target tenant configure a bridge domain to support FCoE traffic. The sample command sequence creates bridge domain b1 under tenant t1 configured to support FCoE connectivity. Example: apic1(config)# tenant t1 apic1(config-tenant)# vrf context v1 apic1(config-tenant-vrf)# exit apic1(config-tenant)# bridge-domain b1 apic1(config-tenant-bd)# fc apic1(config-tenant-bd)# vrf member v1 apic1(config-tenant-bd)# exit apic1(config-tenant)# exit Step 2 Under the same tenant, associate the target EPG with The sample command sequence creates EPG the FCoE-configured bridge domain. e1 and associates that EPG with the FCoE-configured bridge domain b1. Example: apic1(config)# tenant t1 apic1(config-tenant)# application a1 apic1(config-tenant-app)# epg e1 apic1(config-tenant-app-epg)# bridge-domain member b1 apic1(config-tenant-app-epg)# exit apic1(config-tenant-app)# exit apic1(config-tenant)# exit Step 3 Create a VSAN domain, VSAN pools, VLAN pools and VSAN to VLAN mapping. Example: A apic1(config)# vsan-domain dom1 apic1(config-vsan)# vsan 1-10 apic1(config-vsan)# vlan 1-10 apic1(config-vsan)# fcoe vsan 1 vlan 1 loadbalancing src-dst-ox-id apic1(config-vsan)# fcoe vsan 2 vlan 2 In Example A, the sample command sequence creates VSAN domain, dom1 with VSAN pools and VLAN pools, maps VSAN 1 to VLAN 1 and maps VSAN 2 to VLAN 2 In Example B, an alternate sample command sequence creates a reusable VSAN attribute template pol1 and then creates VSAN domain dom1, which inherits the attributes and mappings from that template. Example: B apic1(config)# template vsan-attribute pol1 apic1(config-vsan-attr)# fcoe vsan 2 vlan 12 loadbalancing src-dst-ox-id apic1(config-vsan-attr)# fcoe vsan 3 vlan 13 loadbalancing src-dst-ox-id apic1(config-vsan-attr)# exit apic1(config)# vsan-domain dom1 apic1(config-vsan)# vsan 1-10 apic1(config-vsan)# vlan 1-10 apic1(config-vsan)# inherit vsan-attribute pol1 apic1(config-vsan)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 79 Configuring Layer 2 External Connectivity Configuring FCoE Connectivity Without Policies or Profiles Using the NX-OS Style CLI Command or Action Step 4 Create the physical domain to support the FCoE Initialization (FIP) process. Example: Purpose In the example, the command sequence creates a regular VLAN domain, fipVlanDom, which includes VLAN 120 to support the FIP process. apic1(config)# vlan-domain fipVlanDom apic1(config-vlan)# vlan 120 apic1(config-vlan)# exit Step 5 Under the target tenant configure a regular bridge domain. In the example, the command sequence creates bridge domain fip-bd. Example: apic1(config)# tenant t1 apic1(config-tenant)# vrf context v2 apic1(config-tenant-vrf)# exit apic1(config-tenant)# bridge-domain fip-bd apic1(config-tenant-bd)# vrf member v2 apic1(config-tenant-bd)# exit apic1(config-tenant)# exit Step 6 Under the same tenant, associate this EPG with the configured regular bridge domain. In the example, the command sequence associates EPG epg-fip with bridge domain fip-bd. Example: apic1(config)# tenant t1 apic1(config-tenant)# application a1 apic1(config-tenant-app)# epg epg-fip apic1(config-tenant-app-epg)# bridge-domain member fip-bd apic1(config-tenant-app-epg)# exit apic1(config-tenant-app)# exit apic1(config-tenant)# exit Step 7 Configure a VFC interface with F mode. A In example A the command sequence enables interface 1/2 on leaf switch 101 to function as an F port and associates that interface with VSAN domain dom1. apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/2 apic1(config-leaf-if)# vlan-domain member fipVlanDom apic1(config-leaf-if)# switchport trunk native vlan 120 tenant t1 application a1 epg epg-fip Each of the targeted interfaces must be assigned one (and only one) VSAN in native mode. Each interface may be assigned one or more additional VSANs in regular mode. apic1(config-leaf-if)# exit The sample command sequence associates the target interface 1/2 with: Example: apic1(config-leaf)# exit apic1(config-leaf)# interface vfc 1/2 apic1(config-leaf-if)# switchport mode f apic1(config-leaf-if)# vsan-domain member dom1 apic1(config-leaf-if)# switchport vsan 2 tenant t1 application a1 epg e1 apic1(config-leaf-if)# switchport trunk allowed vsan 3 tenant t1 application a1 epg e2 apic1(config-leaf-if)# exit Example: Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 80 • VLAN 120 for FIP discovery and associates it with EPG epg-fip and application a1 under tenant t1. • VSAN 2 as a native VSAN and associates it with EPG e1 and application a1 under tenant t1. • VSAN 3 as a regular VSAN. Configuring Layer 2 External Connectivity Configuring FCoE Connectivity Without Policies or Profiles Using the NX-OS Style CLI Command or Action Purpose B In example B, the command sequence configures a vFC over a VPC with the same VSAN on both the legs. From the CLI you cannot specify different VSANs on each log. The alternate configuration can be carried out in the APIC advanced GUI. apic1(config)# vpc context leaf 101 102 apic1(config-vpc)# interface vpc vpc1 apic1(config-vpc-if)# vlan-domain member vfdom100 apic1(config-vpc-if)# vsan-domain member dom1 apic1(config-vpc-if)# #For FIP discovery apic1(config-vpc-if)# switchport trunk native vlan 120 tenant t1 application a1 epg epg-fip apic1(config-vpc-if)# switchport vsan 2 tenant t1 application a1 epg e1 apic1(config-vpc-if)# exit apic1(config-vpc)# exit apic1(config)# leaf 101-102 apic1(config-leaf)# interface ethernet 1/3 apic1(config-leaf-if)# channel-group vpc1 vpc apic1(config-leaf-if)# exit apic1(config-leaf)# exit Example: C apic1(config)# leaf 101 apic1(config-leaf)# interface vfc-po pc1 apic1(config-leaf-if)# vsan-domain member dom1 apic1(config-leaf-if)# switchport vsan 2 tenant t1 application a1 epg e1 apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 1/2 apic1(config-leaf-if)# channel-group pc1 apic1(config-leaf-if)# exit apic1(config-leaf)# exit Step 8 Configure a VFC interface with NP mode. Example: apic1(config)# leaf 101 apic1(config-leaf)# interface vfc 1/4 apic1(config-leaf-if)# switchport mode np apic1(config-leaf-if)# vsan-domain member dom1 The sample command sequence enables interface 1/4 on leaf switch 101 to function as an NP port and associates that interface with VSAN domain dom1. Step 9 Assign the targeted FCoE-enabled interfaces a VSAN. Each of the targeted interfaces must be assigned one (and only one) VSAN in native mode. Each interface may be assigned one or Example: apic1(config-leaf-if)# switchport trunk allowed more additional VSANs in regular mode. vsan 1 tenant t1 application a1 epg e1 apic1(config-leaf-if)# switchport vsan 2 tenant t4 application a4 epg e4 The sample command sequence assigns the target interface to VSAN 1 and associates it with EPG e1 and application a1 under tenant t1. "trunk allowed" assigns vsan 1 regular mode status. The command sequence also assigns the interface a required native mode VSAN 2. As this example shows, it is permissible for different VSANs to provide different EPGs running under different tenants access to the same interfaces. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 81 Configuring Layer 2 External Connectivity Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS Style CLI Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS Style CLI The following sample NX-OS style CLI sequences create and use policies to configure FCoE connectivity for EPG e1 under tenant t1. Procedure Command or Action Purpose Step 1 Under the target tenant configure a bridge domain to The sample command sequence creates bridge support FCoE traffic. domain b1 under tenant t1 configured to support FCoE connectivity. Example: apic1# configure apic1(config)# tenant t1 apic1(config-tenant)# vrf context v1 apic1(config-tenant-vrf)# exit apic1(config-tenant)# bridge-domain b1 apic1(config-tenant-bd)# fc apic1(config-tenant-bd)# vrf member v1 apic1(config-tenant-bd)# exit apic1(config-tenant)# exit apic1(config)# Step 2 Under the same tenant, associate your target EPG with The sample command sequence creates EPG the FCoE configured bridge domain. e1 associates that EPG with FCoE-configured bridge domain b1. Example: apic1(config)# tenant t1 apic1(config-tenant)# application a1 apic1(config-tenant-app)# epg e1 apic1(config-tenant-app-epg)# bridge-domain member b1 apic1(config-tenant-app-epg)# exit apic1(config-tenant-app)# exit apic1(config-tenant)# exit apic1(config)# Step 3 Create a VSAN domain, VSAN pools, VLAN pools In Example A, the sample command sequence and VSAN to VLAN mapping. creates VSAN domain, dom1 with VSAN pools and VLAN pools, maps VSAN 1 VLAN 1 and maps VSAN 2 to VLAN 2 Example: A In Example B, an alternate sample command apic1(config)# vsan-domain dom1 sequence creates a reusable vsan attribute apic1(config-vsan)# vsan 1-10 apic1(config-vsan)# vlan 1-10 template pol1 and then creates VSAN domain apic1(config-vsan)# fcoe vsan 1 vlan 1 dom1, which inherits the attributes and loadbalancing mappings from that template. src-dst-ox-id apic1(config-vsan)# fcoe vsan 2 vlan 2 Example: B apic1(config)# template vsan-attribute pol1 apic1(config-vsan-attr)# fcoe vsan 2 vlan 12 loadbalancing src-dst-ox-id Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 82 Configuring Layer 2 External Connectivity Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS Style CLI Command or Action apic1(config-vsan-attr)# Purpose fcoe vsan 3 vlan 13 loadbalancing src-dst-ox-id apic1(config-vsan-attr)# exit apic1(config)# vsan-domain dom1 apic1(config-vsan)# inherit vsan-attribute pol1 apic1(config-vsan)# exit Step 4 Create the physical domain to support the FCoE Initialization (FIP) process. Example: apic1(config)# vlan-domain fipVlanDom apic1(config)# vlan-pool fipVlanPool Step 5 Configure a Fibre Channel SAN policy. Example: apic1# apic1# configure apic1(config)# template fc-fabric-policy ffp1 apic1(config-fc-fabric-policy)# fctimer e-d-tov 1111 apic1(config-fc-fabric-policy)# fctimer r-a-tov 2222 apic1(config-fc-fabric-policy)# fcoe fcmap 0E:FC:01 apic1(config-fc-fabric-policy)# exit Step 6 Create a Fibre Channel node policy. Example: apic1(config)# template fc-leaf-policy flp1 apic1(config-fc-leaf-policy)# fcoe fka-adv-period 44 apic1(config-fc-leaf-policy)# exit Step 7 Create Node Policy Group. Example: apic1(config)# template leaf-policy-group lpg1 apic1(config-leaf-policy-group)# inherit fc-fabric-policy ffp1 apic1(config-leaf-policy-group)# inherit fc-leaf-policy flp1 apic1(config-leaf-policy-group)# exit apic1(config)# exit apic1# Step 8 Create a Node Profile. Example: apic1(config)# leaf-profile lp1 apic1(config-leaf-profile)# leaf-group lg1 apic1(config-leaf-group)# leaf 101 apic1(config-leaf-group)# leaf-policy-group lpg1 The sample command sequence creates Fibre Channel SAN policy ffp1 to specify a combination of error-detect timeout values (EDTOV), resource allocation timeout values (RATOV), and the default FC map values for FCoE-enabled interfaces on a target leaf switch. The sample command sequence creates Fibre Channel node policy flp1 to specify a combination of disruptive load-balancing enablement and FIP keep-alive values. These values also apply to all the FCoE-enabled interfaces on a target leaf switch. The sample command sequence creates a Node Policy group, lpg1, which combines the values of the Fibre Channel SAN policy ffp1 and Fibre Channel node policy, flp1. The combined values of this node policy group can be applied to Node profiles configured later. The sample command sequence creates node profile lp1 associates it with node policy group lpg1, node group lg1, and leaf switch 101. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 83 Configuring Layer 2 External Connectivity Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS Style CLI Command or Action Purpose Step 9 Create an interface policy group for F port interfaces. The sample command sequence creates interface policy group ipg1 and assigns a combination of values that determine priority Example: apic1(config)# template policy-group ipg1 flow control enablement, F port enablement, apic1(config-pol-grp-if)# and slow-drain policy values for any interface priority-flow-control mode auto apic1(config-pol-grp-if)# switchport mode f that this policy group is applied to. apic1(config-pol-grp-if)# slow-drain pause timeout 111 apic1(config-pol-grp-if)# slow-drain congestion-timeout count 55 apic1(config-pol-grp-if)# slow-drain congestion-timeout action log Step 10 Create an interface policy group for NP port interfaces. The sample command sequence creates interface policy group ipg2 and assigns a combination of values that determine priority Example: apic1(config)# template policy-group ipg2 flow control enablement, NP port enablement, apic1(config-pol-grp-if)# and slow-drain policy values for any interface priority-flow-control mode auto apic1(config-pol-grp-if)# switchport mode np that this policy group is applied to. apic1(config-pol-grp-if)# slow-drain pause timeout 111 apic1(config-pol-grp-if)# slow-drain congestion-timeout count 55 apic1(config-pol-grp-if)# slow-drain congestion-timeout action log Step 11 Create an interface profile for F port interfaces. Example: apic1# configure apic1(config)# leaf-interface-profile lip1 apic1(config-leaf-if-profile)# description 'test description lip1' apic1(config-leaf-if-profile)# leaf-interface-group lig1 apic1(config-leaf-if-group)# description 'test description lig1' apic1(config-leaf-if-group)# policy-group ipg1 apic1(config-leaf-if-group)# interface ethernet 1/2-6, 1/9-13 Step 12 Create an interface profile for NP port interfaces. Example: apic1# configure apic1(config)# leaf-interface-profile lip2 apic1(config-leaf-if-profile)# description 'test description lip2' apic1(config-leaf-if-profile)# leaf-interface-group lig2 apic1(config-leaf-if-group)# description 'test description lig2' apic1(config-leaf-if-group)# policy-group ipg2 apic1(config-leaf-if-group)# interface ethernet 1/14 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 84 The sample command sequence creates an interface profile lip1 for F port interfaces, associates the profile with F port specific interface policy group ipg1, and specifies the interfaces to which this profile and its associated policies applies. The sample command sequence creates an interface profile lip2 for NP port interfaces, associates the profile with NP port specific interface policy group ipg2, and specifies the interface to which this profile and its associated policies applies. Configuring Layer 2 External Connectivity Configuring FCoE Over FEX Using NX-OS Style CLI Command or Action Purpose Step 13 Configure QoS Class Policy for Level 1. Example: apic1(config)# qos parameters level1 apic1(config-qos)# pause no-drop cos 3 The sample command sequence specifies the QoS level of FCoE traffic to which priority flow control policy might be applied and pauses no-drop packet handling for Class of Service level 3. Configuring FCoE Over FEX Using NX-OS Style CLI FEX ports are configured as port VSANs. Procedure Step 1 Configure Tenant and VSAN domain: Example: apic1# configure apic1(config)# tenant t1 apic1(config-tenant)# vrf context v1 apic1(config-tenant-vrf)# exit apic1(config-tenant)# bridge-domain b1 apic1(config-tenant-bd)# fc apic1(config-tenant-bd)# vrf member v1 apic1(config-tenant-bd)# exit apic1(config-tenant)# application a1 apic1(config-tenant-app)# epg e1 apic1(config-tenant-app-epg)# bridge-domain member b1 apic1(config-tenant-app-epg)# exit apic1(config-tenant-app)# exit apic1(config-tenant)# exit apic1(config)# vsan-domain dom1 apic1(config-vsan)# vlan 1-100 apic1(config-vsan)# vsan 1-100 apic1(config-vsan)# fcoe vsan 2 vlan 2 loadbalancing src-dst-ox-id apic1(config-vsan)# fcoe vsan 3 vlan 3 loadbalancing src-dst-ox-id apic1(config-vsan)# fcoe vsan 5 vlan 5 apic1(config-vsan)# exit Step 2 Associate FEX to an interface: Example: apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/12 apic1(config-leaf-if)# fex associate 111 apic1(config-leaf-if)# exit Step 3 Configure FCoE over FEX per port, port-channel, and VPC: Example: apic1(config-leaf)# interface vfc 111/1/2 apic1(config-leaf-if)# vsan-domain member dom1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 85 Configuring Layer 2 External Connectivity Verifying FCoE Configuration Using the NX-OS Style CLI apic1(config-leaf-if)# switchport vsan 2 tenant t1 application a1 epg e1 apic1(config-leaf-if)# exit apic1(config-leaf)# interface vfc-po pc1 fex 111 apic1(config-leaf-if)# vsan-domain member dom1 apic1(config-leaf-if)# switchport vsan 2 tenant t1 application a1 epg e1 apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 111/1/3 apic1(config-leaf-if)# channel-group pc1 apic1(config-leaf-if# exit apic1(config-leaf)# exit apic1(config)# vpc domain explicit 12 leaf 101 102 apic1(config-vpc)# exit apic1(config)# vpc context leaf 101 102 apic1(config-vpc)# interface vpc vpc1 fex 111 111 apic1(config-vpc-if)# vsan-domain member dom1 apic1(config-vpc-if)# switchport vsan 2 tenant t1 application a1 epg e1 apic1(config-vpc-if)# exit apic1(config-vpc)# exit apic1(config)# leaf 101-102 apic1(config-leaf)# interface ethernet 1/2 apic1(config-leaf-if)# fex associate 111 apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 111/1/2 apic1(config-leaf-if)# channel-group vpc1 vpc apic1(config-leaf-if)# exit Step 4 Verify the configuration with the following command: Example: apic1(config-vpc)# show vsan-domain detail vsan-domain : dom1 vsan : 1-100 vlan : 1-100 Leaf Interface Vsan Vlan Operational State ------------ ---------------- ---- ----------------------------------------101 vfc111/1/2 2 2 Deployed Vsan-Mode Port-Mode ----------- --------- Native Usage Tenant: t1 App: a1 Epg: e1 101 Deployed PC:pc1 5 5 Native Tenant: t1 App: a1 Epg: e1 101 Deployed vfc111/1/3 3 3 Native F Tenant: t1 App: a1 Epg: e1 Verifying FCoE Configuration Using the NX-OS Style CLI The following show command verifies the FCoE configuration on your leaf switch ports. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 86 Configuring Layer 2 External Connectivity Undeploying FCoE Elements Using the NX-OS Style CLI Procedure Use the show vsan-domain command to verify FCoE is enabled on the target switch. The command example confirms FCoE enabled on the listed leaf switches and its FCF connection details. Example: ifav-isim8-ifc1# show vsan-domain detail vsan-domain : iPostfcoeDomP1 vsan : 1-20 2000 51-52 100-102 104-110 200 1999 3100-3101 3133 vlan : 1-20 2000 51-52 100-102 104-110 200 1999 3100-3101 3133 Leaf ---101 Interface --------vfc1/11 Vsan ---1 Vlan ---1 Vsan Port Mode Mode ------- ---Regular F Usage ---------------Tenant: iPost101 Operational State -----------Deployed App: iPost1 Epg: iPost1 101 vfc1/12 1 1 Regular NP Tenant: iPost101 Deployed App: iPost1 Epg: iPost1 101 PC:infraAccBndl 4 4 Regular Grp_pc01 NP Tenant: iPost101 Deployed App: iPost4 Epg: iPost4 101 vfc1/30 2000 Native Tenant: t1 App: a1 Not deployed (invalid-path) Epg: e1 Undeploying FCoE Elements Using the NX-OS Style CLI Any move to undeploy FCoE connectivity from the ACI fabric requires that you remove the FCoE components on several levels. Procedure Step 1 List the attributes of the leaf port interface, set its mode setting to default, and then remove its EPG deployment and domain association. The example sets the port mode setting of interface vfc 1/2 to default and then removes the deployment of EPG e1 and the association with VSAN Domain dom1 from that interface. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 87 Configuring Layer 2 External Connectivity Undeploying FCoE Elements Using the NX-OS Style CLI Example: apic1(config)# leaf 101 apic1(config-leaf)# interface vfc 1/2 apic1(config-leaf-if)# show run # Command: show running-config leaf 101 interface vfc 1 / 2 # Time: Tue Jul 26 09:41:11 2016 leaf 101 interface vfc 1/2 vsan-domain member dom1 switchport vsan 2 tenant t1 application a1 epg e1 exit exit apic1(config-leaf-if)# no switchport mode apic1(config-leaf-if)# no switchport vsan 2 tenant t1 application a1 epg e1 apic1(config-leaf-if)# no vsan-domain member dom1 apic1(config-leaf-if)# exit apic1(config-leaf)# exit Step 2 List and remove the VSAN mapping and the VLAN and VSAN pools. The example removes the VSAN VLAN mapping for vsan 2, VLAN pool 1-10, and VSAN pool 1-10 from VSAN domain dom1. Example: apic1(config)# vsan-domain dom1 apic1(config-vsan)# show run # Command: show running-config vsan-domain dom1 # Time: Tue Jul 26 09:43:47 2016 vsan-domain dom1 vsan 1-10 vlan 1-10 fcoe vsan 2 vlan 2 exit apic1(config-vsan)# no fcoe vsan 2 apic1(config-vsan)# no vlan 1-10 apic1(config-vsan)# no vsan 1-10 apic1(config-vsan)# exit ################################################################################# NOTE: To remove a template-based VSAN to VLAN mapping use an alternate sequence: ################################################################################# apic1(config)# template vsan-attribute <template_name> apic1(config-vsan-attr)# no fcoe vsan 2 Step 3 Delete the VSAN Domain. The example deletes VSAN domain dom1. Example: apic1(config)# no vsan-domain dom1 Step 4 You can delete the associated tenant, EPG, and selectors if you do not need them. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 88 Configuring Layer 2 External Connectivity Configuring 802.1Q Tunnels Configuring 802.1Q Tunnels About ACI 802.1Q Tunnels Figure 5: ACI 802.1Q Tunnels With Cisco ACI and Cisco APIC Release 2.2(1x) and higher, you can configure 802.1Q tunnels on edge (tunnel) ports to enable point-to-multi-point tunneling of Ethernet frames in the fabric, with Quality of Service (QoS) priority settings. A Dot1q Tunnel transports untagged, 802.1Q tagged, and 802.1ad double-tagged frames as-is across the fabric. Each tunnel carries the traffic from a single customer and is associated with a single bridge domain. ACI front panel ports can be part of a Dot1q Tunnel. Layer 2 switching is done based on Destination MAC (DMAC) and regular MAC learning is done in the tunnel. Edge-port Dot1q Tunnels are supported on second-generation (and later) Cisco Nexus 9000 series switches with "EX" on the end of the switch model name. With Cisco ACI and Cisco APIC Release 2.3(x) and higher, you can also configure multiple 802.1Q tunnels on the same core port to carry double-tagged traffic from multiple customers, each distinguished with an access encapsulation configured for each 802.1Q tunnel. You can also disable MAC Address Learning on 802.1Q tunnels. Both edge ports and core ports can belong to an 802.1Q tunnel with access encapsulation and disabled MAC Address Learning. Both edge ports and core ports in Dot1q Tunnels are supported on third-generation Cisco Nexus 9000 series switches with "FX" on the end of the switch model name. Terms used in this document may be different in the Cisco Nexus 9000 Series documents. Table 8: 802.1Q Tunnel Terminology ACI Documents Cisco Nexus 9000 Series Documents Edge Port Tunnel Port Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 89 Configuring Layer 2 External Connectivity About ACI 802.1Q Tunnels ACI Documents Cisco Nexus 9000 Series Documents Core Port Trunk Port The following guidelines and restrictions apply: • Layer 2 tunneling of VTP, CDP, LACP, LLDP, and STP protocols is supported with the following restrictions: ◦Link Aggregation Control Protocol (LACP) tunneling functions as expected only with point-to-point tunnels using individual leaf interfaces. It is not supported on port-channels (PCs) or virtual port-channels (vPCs). ◦CDP and LLDP tunneling with PCs or vPCs is not deterministic; it depends on the link it chooses as the traffic destination. ◦To use VTP for Layer 2 protocol tunneling, CDP must be enabled on the tunnel. ◦STP is not supported in an 802.1Q tunnel bridge domain when Layer 2 protocol tunneling is enabled and the bridge domain is deployed on Dot1q Tunnel core ports. ◦ACI leaf switches react to STP TCN packets by flushing the end points in the tunnel bridge domain and flooding them in the bridge domain. ◦CDP and LLDP tunneling with more than two interfaces flood packets on all interfaces. ◦With Cisco APIC Release 2.3(x) or higher, the destination MAC address of Layer 2 protocol packets tunneled from edge to core ports is rewritten as 01-00-0c-cd-cd-d0 and the destination MAC address of Layer 2 protocol packets tunneled from core to edge ports is rewritten with the standard default MAC address for the protocol. • If a PC or VPC is the only interface in a Dot1q Tunnel and it is deleted and reconfigured, remove the association of the PC/VPC to the Dot1q Tunnel and reconfigure it. • With Cisco APIC Release 2.2(x) the Ethertypes for double-tagged frames must be 0x9100 followed by 0x8100. However, with Cisco APIC Release 2.3(x) and higher, this limitation no longer applies for edge ports, on third-generation Cisco Nexus switches with "FX" on the end of the switch model name. • For core ports, the Ethertypes for double-tagged frames must be 0x8100 followed by 0x8100. • You can include multiple edge ports and core ports (even across leaf switches) in a Dot1q Tunnel. • An edge port may only be part of one tunnel, but a core port can belong to multiple Dot1q tunnels. • With Cisco APIC Release 2.3(x) and higher, regular EPGs can be deployed on core ports that are used in 802.1Q tunnels. • L3Outs are not supported on interfaces enabled for Dot1q Tunnels. • FEX interfaces are not supported as members of a Dot1q Tunnel. • Interfaces configured as breakout ports do not support 802.1Q tunnels. • Interface-level statistics are supported for interfaces in Dot1q Tunnels, but statistics at the tunnel level are not supported. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 90 Configuring Layer 2 External Connectivity Configuring 802.1Q Tunnels Using the NX-OS Style CLI Configuring 802.1Q Tunnels Using the NX-OS Style CLI Note You can use ports, port-channels, or virtual port channels for interfaces included in a Dot1q Tunnel. Detailed steps are included for configuring ports. See the examples below for the commands to configure edge and core port-channels and virtual port channels. Create a Dot1q Tunnel and configure the interfaces for use in the tunnel using the NX-OS Style CLI, with the following steps: Note Dot1q Tunnels must include 2 or more interfaces. Repeat the steps (or configure two interfaces together), to mark each interface for use in a Dot1q Tunnel. In this example, two interfaces are configured as edge-switch ports, used by a single customer. Use the following steps to configure a Dot1q Tunnel using the NX-OS style CLI: 1 Configure at least two interfaces for use in the tunnel. 2 Create a Dot1q Tunnel. 3 Associate all the interfaces with the tunnel. Before You Begin Configure the tenant that will use the Dot1q Tunnel. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 Configure two interfaces for use in an 802.1Q tunnel, with the following steps: Step 3 leaf ID Identifies the leaf where the interfaces of the Dot1q Tunnel will be located. Example: apic1(config)# leaf 101 Step 4 interface ethernet slot/port Identifies the interface or interfaces to be marked as ports in a tunnel. Example: apic1(config-leaf)# interface ethernet 1/13-14 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 91 Configuring Layer 2 External Connectivity Configuring 802.1Q Tunnels Using the NX-OS Style CLI Step 5 Command or Action Purpose switchport mode dot1q-tunnel {edgePort | corePort} Marks the interfaces for use in an 802.1Q tunnel, and then leaves the configuration mode. Example: The example shows configuring some interfaces for edge port use. Repeat steps 3 to 5 to configure more interfaces for the tunnel. apic1(config-leaf-if)# switchport mode dot1q-tunnel edgePort apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# exit Step 6 Create an 802.1Q tunnel with the following steps: Step 7 leaf ID Returns to the leaf where the interfaces are located. Example: apic1(config)# leaf 101 Step 8 interface ethernetslot/port Returns to the interfaces included in the tunnel. Example: apic1(config-leaf)# interface ethernet 1/13-14 Step 9 switchport tenanttenant-namedot1q-tunnel tunnel-name Associates the interfaces to the tunnel and exits the configuration mode. Example: apic1(config-leaf-if)# switchport tenant tenant64 dot1q-tunnel vrf64_edgetunnel apic1(config-leaf-if)# exit Step 10 Repeat steps 7 to 10 to associate other interfaces with the tunnel. Example: Configuring an 802.1Q Tunnel Using Ports with the NX-OS Style CLI The example marks two ports as edge port interfaces to be used in a Dot1q Tunnel, marks two more ports to be used as core port interfaces, creates the tunnel, and associates the ports with the tunnel. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/13-14 apic1(config-leaf-if)# switchport mode dot1q-tunnel edgePort apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)leaf 102 apic1(config-leaf)# interface ethernet 1/10, 1/21 apic1(config-leaf-if)# switchport mode dot1q-tunnel corePort apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# tenant tenant64 apic1(config-tenant)# dot1q-tunnel vrf64_tunnel apic1(config-tenant-tunnel)# l2protocol-tunnel cdp apic1(config-tenant-tunnel)# l2protocol-tunnel lldp Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 92 Configuring Layer 2 External Connectivity Configuring 802.1Q Tunnels Using the NX-OS Style CLI apic1(config-tenant-tunnel)# apic1(config-tenant-tunnel)# access-encap 200 mac-learning disable apic1(config-tenant-tunnel)# exit apic1(config-tenant)# exit apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/13-14 apic1(config-leaf-if)# switchport tenant tenant64 dot1q-tunnel vrf64_tunnel apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# leaf 102 apic1(config-leaf)# interface ethernet 1/10, 1/21 apic1(config-leaf-if)# switchport tenant tenant64 dot1q-tunnel vrf64_tunnel apic1(config-leaf-if)# exit apic1(config-leaf)# exit Example: Configuring an 802.1Q Tunnel Using Port-Channels with the NX-OS Style CLI The example marks two port-channels as edge-port 802.1Q interfaces, marks two more port-channels as core-port 802.1Q interfaces, creates a Dot1q Tunnel, and associates the port-channels with the tunnel. apic1# configure apic1(config)# tenant tenant64 apic1(config-tenant)# dot1q-tunnel vrf64_tunnel apic1(config-tenant-tunnel)# l2protocol-tunnel cdp apic1(config-tenant-tunnel)# l2protocol-tunnel lldp apic1(config-tenant-tunnel)# access-encap 200 apic1(config-tenant-tunnel)# mac-learning disable apic1(config-tenant-tunnel)# exit apic1(config-tenant)# exit apic1(config)# leaf 101 apic1(config-leaf)# interface port-channel pc1 apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 1/2-3 apic1(config-leaf-if)# channel-group pc1 apic1(config-leaf-if)# exit apic1(config-leaf)# interface port-channel pc1 apic1(config-leaf-if)# switchport mode dot1q-tunnel edgePort apic1(config-leaf-if)# switchport tenant tenant64 dot1q-tunnel vrf64_tunnel apic1(config-tenant-tunnel)# exit apic1(config-tenant)# exit apic1(config)# leaf 102 apic1(config-leaf)# interface port-channel pc2 apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 1/4-5 apic1(config-leaf-if)# channel-group pc2 apic1(config-leaf-if)# exit apic1(config-leaf)# interface port-channel pc2 apic1(config-leaf-if)# switchport mode dot1q-tunnel corePort apic1(config-leaf-if)# switchport tenant tenant64 dot1q-tunnel vrf64_tunnel Example: Configuring an 802.1Q Tunnel Using Virtual Port-Channels with the NX-OS Style CLI The example marks two virtual port-channels (VPCs) as edge-port 802.1Q interfaces for theDot1q Tunnel, marks two more VPCs as core-port interfaces for the tunnel, creates the tunnel, and associates the virtual port-channels with the tunnel. apic1# configure apic1(config)# vpc domain explicit 1 leaf 101 102 apic1(config)# vpc context leaf 101 102 apic1(config-vpc)# interface vpc vpc1 apic1(config-vpc-if)# switchport mode dot1q-tunnel edgePort apic1(config-vpc-if)# exit apic1(config-vpc)# exit apic1(config)# vpc domain explicit 1 leaf 103 104 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 93 Configuring Layer 2 External Connectivity Configuring Dynamic Breakout Ports apic1(config)# vpc context leaf 103 104 apic1(config-vpc)# interface vpc vpc2 apic1(config-vpc-if)# switchport mode dot1q-tunnel corePort apic1(config-vpc-if)# exit apic1(config-vpc)# exit apic1(config)# tenant tenant64 apic1(config-tenant)# dot1q-tunnel vrf64_tunnel apic1(config-tenant-tunnel)# l2protocol-tunnel cdp apic1(config-tenant-tunnel)# l2protocol-tunnel lldp apic1(config-tenant-tunnel)# access-encap 200 apic1(config-tenant-tunnel)# mac-learning disable apic1(config-tenant-tunnel)# exit apic1(config-tenant)# exit apic1(config)# leaf 103 apic1(config-leaf)# interface ethernet 1/6 apic1(config-leaf-if)# channel-group vpc1 vpc apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# leaf 104 apic1(config-leaf)# interface ethernet 1/6 apic1(config-leaf-if)# channel-group vpc1 vpc apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config-vpc)# interface vpc vpc1 apic1(config-vpc-if)# switchport tenant tenant64 dot1q-tunnel vrf64_tunnel apic1(config-vpc-if)# exit Configuring Dynamic Breakout Ports Configuration of Dynamic Breakout Ports To enable a 40 Gigabit Ethernet (GE) leaf switch port to be connected to 4-10GE capable (downlink) devices (connected with Cisco 40-Gigabit to 4X10-Gigabit breakout cables), you configure the 40GE port to breakout (split) to 4-10GE ports. Note This feature is supported only on the access facing ports of the N9K-C9332PQ switch. 100GE breakout ports are currently not supported. Observe the following guidelines and restrictions: • You can configure ports 1 to 26 as downlink ports. Of those ports, breakout ports can be configured on port 1 to 12 and 15 to 26. Ports 13 and 14 do not support breakout. • Breakout subports can be used in the same way other port types in the policy model are used. • When a port is enabled for dynamic breakout, other policies (expect monitoring policies) on the parent port are no longer valid. • When a port is enabled for dynamic breakout, other EPG deployments on the parent port are no longer valid. • A breakout sub-port can not be further broken out using a breakout policy group. You can configure 40GE ports for dynamic breakout using the Basic or Advanced mode APIC GUI, the NX-OS style CLI, or the REST API. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 94 Configuring Layer 2 External Connectivity Configuring Dynamic Breakout Ports Using the NX-OS Style CLI Configuring Dynamic Breakout Ports Using the NX-OS Style CLI Use the following steps to configure a breakout port, verify the configuration, and configure an EPG on a sub port, using the NX-OS style CLI. Before You Begin • The ACI fabric is installed, APIC controllers are online, and the APIC cluster is formed and healthy. • An APIC fabric administrator account is available that will enable creating the necessary fabric infrastructure configurations. • The target leaf switches are registered in the ACI fabric and available. • The 40GE leaf switch ports are connected with Cisco breakout cables to the downlink ports. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf ID Selects the leaf switch where the breakout port will be located and enters leaf configuration mode. Example: apic1(config)# leaf 101 Step 3 interface ethernetslot/port Identifies the interface to be enabled as a 40 Gigabit Ethernet (GE) breakout port. Example: apic1(config-leaf)# interface ethernet 1/16 Step 4 Enables the selected interface for breakout. breakout10g-4x Note Example: apic1(config-leaf-if)# breakout 10g-4x Step 5 The 25g-4x keyword for enabling 100GE ports for breakout is not supported at this time. Verifies the configuration by showing the running configuration of the interface and returns to global configuration mode. show run Example: apic1(config-leaf-if)# show run # Command: show running-config leaf 101 interface ethernet 1 / 16 # Time: Fri Dec 2 18:13:39 2016 leaf 101 interface ethernet 1/16 breakout 10g-4x apic1(config-leaf-if)# exit apic1(config-leaf)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 95 Configuring Layer 2 External Connectivity Configuring Dynamic Breakout Ports Using the NX-OS Style CLI Step 6 Command or Action Purpose tenant tenant-name Selects or creates the tenant that will consume the breakout ports and enters tenant configuration mode. Example: apic1(config)# tenant tenant64 Step 7 vrf context vrf-name Example: Creates or identifies the Virtual Routing and Forwarding (VRF) instance associated with the tenant and exits the configuration mode. apic1(config-tenant)# vrf context vrf64 apic1(config-tenant-vrf)# exit Step 8 bridge-domain bridge-domain-name Creates or identifies the bridge-domain associated with the tenant and enters BD configuration mode. Example: apic1(config-tenant)# bridge-domain bd64 Step 9 vrf member vrf-name Associates the VRF with the bridge-domain and exits the configuration mode. Example: apic1(config-tenant-bd)# vrf member vrf64 apic1(config-tenant-bd)# exit Step 10 application application-profile-name Creates or identifies the application profile associated with the tenant and the EPG. Example: apic1(config-tenant)# application app64 Step 11 epg epg-name Creates or identifies the EPG and enters into EPG configuration mode. Example: apic1(config-tenant)# epg epg64 Step 12 bridge-domain member bridge-domain-name Associates the EPG with the bridge domain and returns to global configuration mode. Example: Configure the sub ports as desired, for example, use the speed command in leaf interface mode to configure a sub port. apic1(config-tenant-app-epg)# bridge-domain member bd64 apic1(config-tenant-app-epg)# exit apic1(config-tenant-app)# exit apic1(config-tenant)# exit Step 13 speed interface-speed Example: apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/16/1 apic1(config-leaf-if)# speed 10G apic1(config-leaf-if)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 96 Enters leaf interface mode, sets the speed of an interface, and exits the configuration mode. Configuring Layer 2 External Connectivity Configuring Dynamic Breakout Ports Using the NX-OS Style CLI Step 14 Command or Action Purpose show run After you have configured the sub ports, entering this command in leaf configuration mode displays the sub port details. Example: apic1(config-leaf)# show run The port on leaf 101 at interface 1/16 is confirmed enabled for breakout with sub ports 1/16/1, 1/16/2, 1/16/3, and 1/16/4. This example configures the port for breakout: apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/16 apic1(config-leaf-if)# breakout 10g-4x This example configures the EPG for the sub ports. apic1(config)# tenant tenant64 apic1(config-tenant)# vrf context vrf64 apic1(config-tenant-vrf)# exit apic1(config-tenant)# bridge-domain bd64 apic1(config-tenant-bd)# vrf member vrf64 apic1(config-tenant-bd)# exit apic1(config-tenant)# application app64 apic1(config-tenant-app)# epg epg64 apic1(config-tenant-app-epg)# bridge-domain member bd64 apic1(config-tenant-app-epg)# end This example sets the speed for the breakout sub ports to 10G. apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/16/1 apic1(config-leaf-if)# speed 10G apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 1/16/2 apic1(config-leaf-if)# speed 10G apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 1/16/3 apic1(config-leaf-if)# speed 10G apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 1/16/4 apic1(config-leaf-if)# speed 10G apic1(config-leaf-if)# exit This example shows the four sub ports connected to leaf 101, interface 1/16. apic1#(config-leaf)# show run # Command: show running-config leaf 101 # Time: Fri Dec 2 00:51:08 2016 leaf 101 interface ethernet 1/16/1 speed 10G negotiate auto link debounce time 100 exit interface ethernet 1/16/2 speed 10G negotiate auto link debounce time 100 exit interface ethernet 1/16/3 speed 10G negotiate auto link debounce time 100 exit interface ethernet 1/16/4 speed 10G Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 97 Configuring Layer 2 External Connectivity Microsegmentation on Virtual Switches negotiate auto link debounce time 100 exit interface ethernet 1/16 breakout 10g-4x exit interface vfc 1/16 Microsegmentation on Virtual Switches Configuring Microsegmentation on Virtual Switches Microsegmentation with the Cisco Application Centric Infrastructure (ACI) provides the ability to automatically assign endpoints to logical security zones called endpoint groups (EPGs) based on various network-based or virtual machine (VM)-based attributes. This section contains instructions for configuring microsegment (uSeg) EPGs on virtual switches. Microsegmentation with Cisco ACI provides support for virtual endpoints attached to the following: • VMware vSphere Distributed Switch (VDS) • Cisco Application Virtual Switch (AVS) • Microsoft vSwitch See the Cisco ACI Virtualization Guide for information about how Microsegmentation with Cisco ACI works, prerequisites, guidelines, and scenarios. Configuring Microsegmentation with Cisco ACI Using the NX-OS-style CLI This section describes how to configure Microsegmentation with Cisco ACI for Cisco AVS, VMware VDS or Microsoft vSwitch using VM-based attributes within an application EPG. Procedure Step 1 In the CLI, enter configuration mode: Example: apic1# configure apic1(config)# Step 2 Create the uSeg EPG: Example: This example is for an application EPG. Note The command to allow microsegmentation in the following example is required for VMware VDS only. apic1(config)# tenant cli-ten1 apic1(config-tenant)# application cli-a1 apic1(config-tenant-app)# epg cli-baseEPG1 apic1(config-tenant-app-epg)# bridge-domain member cli-bd1 apic1(config-tenant-app-epg)# vmware-domain member cli-vmm1 allow-micro-segmentation Example: Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 98 Configuring Layer 2 External Connectivity Configuring Microsegmentation on Virtual Switches (Optional) This example sets match EPG precedence for the uSeg EPG: apic1(config)# tenant Coke apic1(config-tenant)# application cli-a1 apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1 apic1(config-tenant-app-uepg)# match-precedence 10 Example: This example uses a filter based on the attribute VM Name. apic1(config)# tenant cli-ten1 apic1(config-tenant)# application cli-a1 apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1 apic1(config-tenant-app-uepg)# attribute-logical-expression ‘vm-name contains <cos1>’ Example: This example uses a filter based on an IP address. apic1(config)# tenant cli-ten1 apic1(config-tenant)# application cli-a1 apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1 apic1(config-tenant-app-uepg)# attribute-logical-expression ‘ip equals <FF:FF:FF:FF:FF:FF>’ Example: This example uses a filter based on a MAC address. apic1(config)# tenant cli-ten1 apic1(config-tenant)# application cli-a1 apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1 apic1(config-tenant-app-uepg)# attribute-logical-expression ‘mac equals <FF-FF-FF-FF-FF-FF>’ Example: This example uses the operator AND to match all attributes and the operator OR to match any attribute. apic1(config)# tenant cli-ten1 apic1(config-tenant)# application cli-a1 apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented apic1(config-tenant-app-uepg)# attribute-logical-expression 'hv equals host-123 OR (guest-os equals "Ubuntu Linux (64-bit)" AND domain contains fex)' Step 3 Verify the uSeg EPG creation: Example: The following example is for a uSeg EPG with a VM name attribute filter apic1(config-tenant-app-uepg)# show running-config # Command: show running-config tenant cli-ten1 application cli-a1 epg cli-uepg1 type micro-segmented # Time: Thu Oct 8 11:54:32 2015 tenant cli-ten1 application cli-a1 epg cli-uepg1 type micro-segmented bridge-domain cli-bd1 attribute-logical-expression ‘vm-name contains cos1 force’ {vmware-domain | microsoft-domain} member cli-vmm1 exit exit exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 99 Configuring Layer 2 External Connectivity Configuring Microsegmentation on Bare-Metal Configuring Microsegmentation on Bare-Metal Using Microsegmentation with Network-based Attributes on Bare Metal You can use Cisco APIC to configure Microsegmentation with Cisco ACI to create a new, attribute-based EPG using a network-based attribute, a MAC address or one or more IP addresses. You can configure Microsegmentation with Cisco ACI using network-based attributes to isolate VMs or physical endpoints within a single base EPG or VMs or physical endpoints in different EPGs. Using an IP-based Attribute You can use an IP-based filter to isolate a single IP address, a subnet, or multiple of noncontiguous IP addresses in a single microsegment. You might want to isolate physical endpoints based on IP addresses as a quick and simply way to create a security zone, similar to using a firewall. Using a MAC-based Attribute You can use a MAC-based filter to isolate a single MAC address or multiple MAC addresses. You might want to do this if you have a server sending bad traffic int he network. By creating a microsegment with a MAC-based filter, you can isolate the server. Configuring a Network-Based Microsegmented EPG in a Bare-Metal Environment Using the NX-OS Style CLI This section describes how to configure microsegmentation with Cisco ACI using network-based attributes (IP address or MAC address) within a base EPG in a bare-metal environment. Procedure Command or Action Step 1 In the CLI, enter configuration mode: Example: apic1# configure apic1(config)# Step 2 Create the microsegment: Example: This example uses a filter based on an IP address. apic1(config)# tenant cli-ten1 apic1(config-tenant)# application cli-a1 apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1 apic1(config-tenant-app-uepg)# attribute cli-upg-att match ip <X.X.X.X> #Schemes to express the ip A.B.C.D IP Address A.B.C.D/LEN IP Address and mask Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 100 Purpose Configuring Layer 2 External Connectivity Configuring a Network-Based Microsegmented EPG in a Bare-Metal Environment Using the NX-OS Style CLI Command or Action Purpose Example: This example uses a filter based on a MAC address. apic1(config)# tenant cli-ten1 apic1(config-tenant)# application cli-a1 apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1 apic1(config-tenant-app-uepg)# attribute cli-upg-att match mac <FF-FF-FF-FF-FF-FF> #Schemes to express the mac E.E.E MAC address (Option 1) EE-EE-EE-EE-EE-EE MAC address (Option 2) EE:EE:EE:EE:EE:EE MAC address (Option 3) EEEE.EEEE.EEEE MAC address (Option 4) Example: This example uses a filter based on a MAC address and enforces intra-EPG isolation between all members of this uSeg EPG: apic1(config)# tenant cli-ten1 apic1(config-tenant)# application cli-a1 apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented apic1(config-tenant-app-uepg)# isolation enforced apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1 apic1(config-tenant-app-uepg)# attribute cli-upg-att match mac <FF-FF-FF-FF-FF-FF> #Schemes to express the mac E.E.E MAC address (Option 1) EE-EE-EE-EE-EE-EE MAC address (Option 2) EE:EE:EE:EE:EE:EE MAC address (Option 3) EEEE.EEEE.EEEE MAC address (Option 4) Step 3 Deploy the EPG. Example: This example deploys the EPG and bids to the leaf. apic1(config)# leaf 101 apic1(config-leaf)# deploy-epg tenant cli-ten1 application cli-a1 epg cli-uepg1 type micro-segmented Step 4 Verify the microsegment creation: Example: apic1(config-tenant-app-uepg)# show running-config # Command: show running-config tenant cli-ten1 application cli-app1 epg cli-uepg1 type micro-segmented # Time: Thu Oct 8 11:54:32 2015 tenant cli-ten1 application cli-app1 epg cli-esx1bu type micro-segmented bridge-domain cli-bd1 attribute cli-uepg-att match mac 00:11:22:33:44:55 exit exit exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 101 Configuring Layer 2 External Connectivity Configuring Layer 2 IGMP Snoop Multicast Configuring Layer 2 IGMP Snoop Multicast About Cisco APIC and IGMP Snooping IGMP snooping is the process of listening to Internet Group Management Protocol (IGMP) network traffic. The feature allows a network switch to listen in on the IGMP conversation between hosts and routers and filter multicasts links that do not need them, thus controlling which ports receive specific multicast traffic. Cisco APIC provides support for the full IGMP snooping feature included on a traditional switch such as the N9000 standalone. • Policy-based IGMP snooping configuration per bridge domain APIC enables you to configure a policy in which you enable, disable, or customize the properties of IGMP Snooping on a per bridge-domain basis. You can then apply that policy to one or multiple bridge domains. • Static port group implementation IGMP static port grouping enables you to pre-provision ports, already statically-assigned to an application EPG, as the switch ports to receive and process IGMP multicast traffic. This pre-provisioning prevents the join latency which normally occurs when the IGMP snooping stack learns ports dynamically. Static group membership can be pre-provisioned only on static ports (also called, static-binding ports) assigned to an application EPG. • Access group configuration for application EPGs An “access-group” is used to control what streams can be joined behind a given port. An access-group configuration can be applied on interfaces that are statically assigned to an application EPG in order to ensure that the configuration can be applied on ports that will actually belong to the that EPG. Only Route-map-based access groups are allowed. Note You can use vzAny to enable protocols such as IGMP Snooping for all the EPGs in a VRF. For more information about vzAny, see Use vzAny to Automatically Apply Communication Rules to all EPGs in a VRF. To use vzAny, navigate to Tenants > tenant-name > Networking > VRFs > vrf-name > EPG Collection for VRF. Enabling IGMP Snooping Static Port Groups IGMP static port grouping enables you to pre-provision ports, that were previously statically-assigned to an application EPG, to enable the switch ports to receive and process IGMP multicast traffic. This pre-provisioning prevents the join latency which normally occurs when the IGMP snooping stack learns ports dynamically. Static group membership can be pre-provisioned only on static ports assigned to an application EPG. Static group membership can be configured through the APIC GUI, CLI, and REST API interfaces. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 102 Configuring Layer 2 External Connectivity Configuring and Assigning an IGMP Snooping Policy to a Bridge Domain using the NX-OS Style CLI Configuring and Assigning an IGMP Snooping Policy to a Bridge Domain using the NX-OS Style CLI Before You Begin • Create the tenant that will consume the IGMP Snooping policy. • Create the bridge domain for the tenant, where you will attach he IGMP Snooping policy. Procedure Command or Action Purpose Step 1 Create a snooping policy based on default values. Example: apic1(config-tenant)# template ip igmp snooping policy cookieCut1 apic1(config-tenant-template-ip-igmp-snooping)# show run all # Command: show running -config all tenant foo template ip igmp snooping policy cookieCut1 # Time: Thu Oct 13 18:26:03 2016 tenant t_10 template ip igmp snooping policy cookieCut1 ip igmp snooping no ip igmp snooping fast-leave ip igmp snooping last-member-query-interval 1 no ip igmp snooping querier ip igmp snooping query-interval 125 ip igmp snooping query-max-response-time 10 ip igmp snooping stqrtup-query-count 2 ip igmp snooping startup-query-interval 31 no description exit exit apic1(config-tenant-template-ip-igmp-snooping)# Step 2 Modify the snooping policy as necessary. Example: apic1(config-tenant-template-ip-igmp-snooping)# ip igmp snooping query-interval 300 apic1(config-tenant-template-ip-igmp-snooping)# show run all # Command: show running -config all tenant foo template ip igmp snooping policy cookieCut1 #Time: Thu Oct 13 18:26:03 2016 tenant foo template ip igmp snooping policy cookieCut1 ip igmp snooping no ip igmp snooping fast-leave ip igmp snooping last-member-query-interval 1 no ip igmp snooping querier ip igmp snooping query-interval 300 The example NX-OS style CLI sequence: • Creates an IGMP Snooping policy named cookieCut1 with default values. • Displays the default IGMP Snooping values for the policy cookieCut1. The example NX-OS style CLI sequence: • Specifies a custom value for the query-interval value in the IGMP Snooping policy named cookieCut1. • Confirms the modified IGMP Snooping value for the policy cookieCut1. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 103 Configuring Layer 2 External Connectivity Enabling IGMP Snooping and Multicast on Static Ports in the NX-OS Style CLI Command or Action Purpose ip igmp snooping query-max-response-time 10 ip igmp snooping stqrtup-query-count 2 ip igmp snooping startup-query-interval 31 no description exit exit apic1(config-tenant-template-ip-igmp-snooping)# exit apic1(config--tenant)# Step 3 Assign the policy to a bridge domain. The example NX-OS style CLI sequence: Example: apic1(config-tenant)# int bridge-domain bd3 apic1(config-tenant-interface)# ip igmp snooping policy cookieCut1 • Navigates to bridge domain, BD3. for the query-interval value in the IGMP Snooping policy named cookieCut1. • Assigns the IGMP Snooping policy with a modified IGMP Snooping value for the policy cookieCut1. What to Do Next You can assign the IGMP Snooping policy to multiple bridge domains. Enabling IGMP Snooping and Multicast on Static Ports in the NX-OS Style CLI You can enable IGMP snooping and multicast on ports that have been statically assigned to an EPG. Then you can create and assign access groups of users that are permitted or denied access to the IGMP snooping and multicast traffic enabled on those ports. The steps described in this task assume the pre-configuration of the following entities: • Tenant: tenant_A • Application: application_A • EPG: epg_A • Bridge Domain: bridge_domain_A • vrf: vrf_A -- a member of bridge_domain_A • VLAN Domain: vd_A (configured with a range of 300-310) • Leaf switch: 101 and interface 1/10 The target interface 1/10 on switch 101 is associated with VLAN 305 and statically linked with tenant_A, application_A, epg_A • Leaf switch: 101 and interface 1/11 The target interface 1/11 on switch 101 is associated with VLAN 309 and statically linked with tenant_A, application_A, epg_A Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 104 Configuring Layer 2 External Connectivity Enabling IGMP Snoop Access Groups Before You Begin Before you begin to enable IGMP snooping and multicasting for an EPG, complete the following tasks. • Identify the interfaces to enable this function and statically assign them to that EPG Note For details on static port assignment, see Deploying an EPG on a Specific Port with APIC Using the NX-OS Style CLI in Cisco APIC Layer 3 Configuration Guide. • Identify the IP addresses that you want to be recipients of IGMP snooping multicast traffic. Procedure Command or Action Purpose Step 1 On the target interfaces enable IGMP snooping and layer 2 The example sequences enable: multicasting • IGMP snooping on the statically-linked target interface 1/10 Example: and associates it with a multicast IP apic1# conf t address, 225.1.1.1 apic1(config)# tenant tenant_A apic1(config-tenant)# application application_A apic1(config-tenant-app)# epg epg_A apic1(config-tenant-app-epg)# ip igmp snooping static-group 225.1.1.1 leaf 101 interface ethernet 1/10 vlan 305 apic1(config-tenant-app-epg)# end • IGMP snooping on the statically-linked target interface 1/11 and associates it with a multicast IP address, 227.1.1.1 apic1# conf t apic1(config)# tenant tenant_A; application application_A; epg epg_A apic1(config-tenant-app-epg)# ip igmp snooping static-group 227.1.1.1 leaf 101 interface ethernet 1/11 vlan 309 apic1(config-tenant-app-epg)# exit apic1(config-tenant-app)# exit Enabling IGMP Snoop Access Groups An “access-group” is used to control what streams can be joined behind a given port. An access-group configuration can be applied on interfaces that are statically assigned to an application EPG in order to ensure that the configuration can be applied on ports that will actually belong to the that EPG. Only Route-map-based access groups are allowed. IGMP snoop access groups can be configured through the APIC GUI, CLI, and REST API interfaces. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 105 Configuring Layer 2 External Connectivity Enabling Group Access to IGMP Snooping and Multicast using the NX-OS Style CLI Enabling Group Access to IGMP Snooping and Multicast using the NX-OS Style CLI After you have enabled IGMP snooping and multicast on ports that have been statically assigned to an EPG, you can then create and assign access groups of users that are permitted or denied access to the IGMP snooping and multicast traffic enabled on those ports. The steps described in this task assume the pre-configuration of the following entities: • Tenant: tenant_A • Application: application_A • EPG: epg_A • Bridge Domain: bridge_domain_A • vrf: vrf_A -- a member of bridge_domain_A • VLAN Domain: vd_A (configured with a range of 300-310) • Leaf switch: 101 and interface 1/10 The target interface 1/10 on switch 101 is associated with VLAN 305 and statically linked with tenant_A, application_A, epg_A • Leaf switch: 101 and interface 1/11 The target interface 1/11 on switch 101 is associated with VLAN 309 and statically linked with tenant_A, application_A, epg_A Note For details on static port assignment, see Deploying an EPG on a Specific Port with APIC Using the NX-OS Style CLI in Cisco APIC Layer 2 Configuration Guide. Procedure Command or Action Step 1 Define the route-map "access groups." Example: apic1# conf t apic1(config)# tenant tenant_A; application application_A; epg epg_A apic1(config-tenant)# route-map fooBroker permit apic1(config-tenant-rtmap)# match ip multicast group 225.1.1.1/24 apic1(config-tenant-rtmap)# exit apic1(config-tenant)# route-map fooBroker deny apic1(config-tenant-rtmap)# match ip multicast group 227.1.1.1/24 apic1(config-tenant-rtmap)# exit Step 2 Verify route map configurations. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 106 Purpose The example sequences configure: • Route-map-access group "foobroker" linked to multicast group 225.1.1.1/24, access permited • Route-map-access group "foobroker" linked to multicast group 227.1.1.1/24, access denied Configuring Layer 2 External Connectivity Enabling Group Access to IGMP Snooping and Multicast using the NX-OS Style CLI Command or Action Purpose Example: apic1(config-tenant)# show running-config tenant test route-map fooBroker # Command: show running-config tenant test route-map fooBroker # Time: Mon Aug 29 14:34:30 2016 tenant test route-map fooBroker permit 10 match ip multicast group 225.1.1.1/24 exit route-map fooBroker deny 20 match ip multicast group 227.1.1.1/24 exit exit Step 3 Specify the access group connection path. The example sequences configure: Example: apic1(config-tenant)# application application_A apic1(config-tenant-app)# epg epg_A apic1(config-tenant-app-epg)# ip igmp snooping access-group route-map fooBroker leaf 101 interface ethernet 1/10 vlan 305 apic1(config-tenant-app-epg)# ip igmp snooping access-group route-map newBroker leaf 101 interface ethernet 1/10 vlan 305 • Route-map-access group "foobroker" connected through leaf switch 101, interface 1/10, and VLAN 305. • Route-map-access group "newbroker" connected through leaf switch 101, interface 1/10, and VLAN 305. Step 4 Verify the access group connections. Example: apic1(config-tenant-app-epg)# show run # Command: show running-config tenant tenant_A application application_A epg epg_A # Time: Mon Aug 29 14:43:02 2016 tenant tenent_A application application_A epg epg_A bridge-domain member bridge_domain_A ip igmp snooping access-group fooBroker leaf 101 interface ethernet ip igmp snooping access-group fooBroker leaf 101 interface ethernet ip igmp snooping access-group newBroker leaf 101 interface ethernet ip igmp snooping static-group 101 interface ethernet 1/10 vlan 305 ip igmp snooping static-group 101 interface ethernet 1/11 vlan 309 exit exit exit route-map 1/10 vlan route-map 1/11 vlan route-map 1/10 vlan 225.1.1.1 305 309 305 leaf 225.1.1.1 leaf Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 107 Configuring Layer 2 External Connectivity Deploying an EPG on a Specific Port with APIC Using the NX-OS Style CLI Deploying an EPG on a Specific Port with APIC Using the NX-OS Style CLI Procedure Step 1 Configure a VLAN domain: Example: apic1(config)# vlan-domain dom1 apic1(config-vlan)# vlan 10-100 Step 2 Create a tenant: Example: apic1# configure apic1(config)# tenant t1 Step 3 Create a private network/VRF: Example: apic1(config-tenant)# vrf context ctx1 apic1(config-tenant-vrf)# exit Step 4 Create a bridge domain: Example: apic1(config-tenant)# bridge-domain bd1 apic1(config-tenant-bd)# vrf member ctx1 apic1(config-tenant-bd)# exit Step 5 Create an application profile and an application EPG: Example: apic1(config-tenant)# application AP1 apic1(config-tenant-app)# epg EPG1 apic1(config-tenant-app-epg)# bridge-domain member bd1 apic1(config-tenant-app-epg)# exit apic1(config-tenant-app)# exit apic1(config-tenant)# exit Step 6 Associate the EPG with a specific port: Example: apic1(config)# leaf 1017 apic1(config-leaf)# interface ethernet 1/13 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk allowed vlan 20 tenant t1 application AP1 epg EPG1 Note The vlan-domain and vlan-domain member commands mentioned in the above example are a pre-requisite for deploying an EPG on a port. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 108 Configuring Layer 2 External Connectivity Configuring Port Security Configuring Port Security About Port Security and ACI The port security feature protects the ACI fabric from being flooded with unknown MAC addresses by limiting the number of MAC addresses learned per port. The port security feature support is available for physical ports, port channels, and virtual port channels. Port Security Guidelines and Restrictions The guidelines and restrictions are as follows: • Port security is available per port. • Port security is supported for physical ports, port channels, and virtual port channels (vPCs). • Static and dynamic MAC addresses are supported. • MAC address moves are supported from secured to unsecured ports and from unsecured ports to secured ports. • The MAC address limit is enforced only on the MAC address and is not enforced on a MAC and IP address. • Port security is not supported with the Fabric Extender (FEX). Port Security at Port Level In the APIC, the user can configure the port security on switch ports. Once the MAC limit has exceeded the maximum configured value on a port, all traffic from the exceeded MAC addresses is forwarded. The following attributes are supported: • Port Security Timeout—The current supported range for the timeout value is from 60 to 3600 seconds. • Violation Action—The violation action is available in protect mode. In the protect mode, MAC learning is disabled and MAC addresses are not added to the CAM table. Mac learning is re-enabled after the configured timeout value. • Maximum Endpoints—The current supported range for the maximum endpoints configured value is from 0 to 12000. If the maximum endpoints value is 0, the port security policy is disabled on that port. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 109 Configuring Layer 2 External Connectivity Configuring a Port Security Policy Group Template Configuring a Port Security Policy Group Template Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# Step 2 configure [no] template policy-group policy-group-name Creates (or deletes) a policy group template. Example: apic1(config)# template policy-group PortSecGrp1 Step 3 [no] switchport access vlan vlan-id tenant tenant-name application application-name epg epg-name Example: apic1(config-pol-grp-if)# switchport access vlan 4 tenant ExampleCorp application Web epg webEpg Step 4 [no] switchport port-security maximum number-of-addresses Sets the maximum number of secure MAC addresses for the port. The range is 0 to 12000 addresses. The default is 1 address. Example: apic1(config-pol-grp-if)# switchport port-security maximum 1 Step 5 [no] switchport port-security violation protect Example: switchport port-security violation protect apic1(config-pol-grp-if)# Step 6 Sets the action to be taken when a security violation is detected. The protect action drops packets with unknown source addresses until you remove a sufficient number of secure MAC addresses to drop below the maximum value. Returns to global configuration mode. exit Example: apic1(config-pol-grp-if)# exit Example This example shows how to create a port security policy group template. apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 110 Configuring Layer 2 External Connectivity Configuring Port Security on an Interface Using a Template apic1(config)# template policy-group PortSecGrp1 apic1(config-pol-grp-if)# switchport port-security maximum 20 apic1(config-pol-grp-if)# switchport port-security violation protect apic1(config-pol-grp-if)# exit What to Do Next Apply the port security template to an interface. Configuring Port Security on an Interface Using a Template Before You Begin Create a port security policy group template. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 interface type-or-range Specifies a port or a range of ports to be configure. Example: apic1(config-leaf)# interface eth 1/2-4 Step 4 [no] policy-group policy-group-name Applies the policy group template to the port or range of ports. Example: apic1(config-leaf-if)# policy-group PortSecGrp1 Example This example shows how to apply a port security policy group template to a range of Ethernet ports. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/2-4 apic1(config-leaf-if)# policy-group PortSecGrp1 This example shows how to configure port security on a port channel using a template. apic1# configure apic1(config)# template port-channel po1 apic1(config-if)# switchport port-security maximum 10 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 111 Configuring Layer 2 External Connectivity Configuring Port Security on an Interface Using Overrides apic1(config-if)# switchport port-security violation protect apic1(config-if)# exit apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/3-4 apic1(config-leaf-if)# channel-group po1 apic1(config-leaf-if)# exit apic1(config-leaf)# exit Configuring Port Security on an Interface Using Overrides Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 interface type-or-range Specifies an interface or a range of interfaces to be configured. Example: apic1(config-leaf)# interface eth 1/2-4 Step 4 [no] switchport port-security maximum number-of-addresses Sets the maximum number of secure MAC addresses for the interface. The range is 0 to 12000 addresses. The default is 1 address. Example: apic1(config-leaf-if)# switchport port-security maximum 1 Step 5 [no] switchport port-security violation protect Example: apic1(config-leaf-if)# switchport port-security violation protect Sets the action to be taken when a security violation is detected. The protect action drops packets with unknown source addresses until you remove a sufficient number of secure MAC addresses to drop below the maximum value. Example This example shows how to configure port security on an Ethernet interface. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/2 apic1(config-leaf-if)# switchport port-security maximum 10 apic1(config-leaf-if)# switchport port-security violation protect Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 112 Configuring Layer 2 External Connectivity 802.1x Port and Node Authentication This example shows how to configure port security on a port channel. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface port-channel po2 apic1(config-leaf-if)# switchport port-security maximum 10 apic1(config-leaf-if)# switchport port-security violation protect This example shows how to configure port security on a virtual port channel (VPC). apic1# configure apic1(config)# vpc domain explicit 1 leaf 101 102 apic1(config-vpc)# exit apic1(config)# template port-channel po4 apic1(config-if)# exit apic1(config)# leaf 101-102 apic1(config-leaf)# interface eth 1/11-12 apic1(config-leaf-if)# channel-group po4 vpc apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# vpc context leaf 101 102 apic1(config-vpc)# interface vpc po4 apic1(config-vpc-if)# switchport port-security maximum 10 apic1(config-vpc-if)# switchport port-security violation protect 802.1x Port and Node Authentication 802.1x Port and Node Authentication IEEE 802.1x is a port-based authentication mechanism to prevent unauthorized devices from gaining access to the network. You can configure 802.1x port and node authentication using the NX-OS style CLI. Configuring a Port Authentication Policy Procedure Step 1 In the CLI, enter configuration mode: Example: apic1# configure apic1(config)# Step 2 Create a policy group: Example: apic1(config)# template policy-group mypol Step 3 Configure port-level authentication policy in the policy group: Example: apic1(config-pol-grp-if)# switchport port-authentication mydot1x Step 4 Configure host mode (two modes are supported: multi-host and single-host - single being the default setting): Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 113 Configuring Layer 2 External Connectivity 802.1x Port and Node Authentication Example: apic1(config-port-authentication)# host-mode multi-host Step 5 Enable this policy (policy is disabled by default): Example: apic1(config-port-authentication)# no shutdown apic1(config-port-authentication)# exit apic1(config-pol-grp-if)# exit apic1(config)# Step 6 Configure the leaf interface profile: Example: apic1(config)#leaf-interface-profile myprofile Step 7 Configure a policy group for the leaf switch interface profile: Example: apic1(config-leaf-if-profile)#leaf-interface-group mygroup Step 8 Specify ports and/or interfaces for your interface group: Example: apic1(config-leaf-if-group)# interface ethernet 1/10-12 Step 9 Apply the policy on your interface group: Example: apic1(config-leaf-if-group)# policy-group mypol apic1(config-leaf-if-group)# exit apic1(config-leaf-if-profile)# exit Step 10 Configure the leaf profile : Example: apic1(config)# apic1(config)# leaf-profile myleafprofile Step 11 Configure the leaf policy group and specify leaf switch nodes for the group: Example: apic1(config-leaf-profile)# leaf-group myleafgrp apic1(config-leaf-group)# leaf 101 apic1(config-leaf-group)# exit Step 12 Apply an interface policy on the leaf switch profile: Example: apic1(config-leaf-profile)# leaf-interface-profile myprofile apic1(config-leaf-group)# exit apic1(config)# Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 114 Configuring Layer 2 External Connectivity 802.1x Port and Node Authentication Configuring a Node Authentication Policy Procedure Step 1 In the CLI, enter configuration mode: Example: apic1# configure apic1(config)# Step 2 Configure the radius authentication group: Example: apic1(config)# aaa group server radius myradiusgrp apic1(config-radius)#server 192.168.0.100 priority 1 apic1(config-radius)#exit Step 3 Configure node level port authentication policy: Example: apic1(config)# policy-map type port-authentication mydot1x apic1(config-pmap-port-authentication)#radius-provider-group myradiusgrp Step 4 [Optional] Override the defaul VLAN ID if authentication fails. : Example: apic1(config-pmap-port-authentication)#fail-auth-vlan 2001 Step 5 [Optional] Override defaul EPG if authentication fails: Example: apic1(config-pmap-port-authentication)#fail-auth-epg tenant tn1 application ap1 epg epg256 apic1(config)# exit Step 6 Configure policy group and specify port authentication policy in the group: Example: apic1(config)#template leaf-policy-group lpg2 apic1(config-leaf-policy-group)# port-authentication mydot1x apic1(config-leaf-policy-group)#exit Step 7 Configure the leaf switch profile: Example: apic1(config)# leaf-profile mylp2 Step 8 Configure a group for the leaf switch profile and specify the policy group: Example: apic1(config-leaf-profile)#leaf-group mylg2 apic1(config-leaf-group)# leaf-policy-group lpg2 apic1(config-leaf-group)#exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 115 Configuring Layer 2 External Connectivity Configuring Proxy ARP Configuring Proxy ARP About Proxy ARP Proxy ARP in Cisco ACI enables endpoints within a network or subnet to communicate with other endpoints without knowing the real MAC address of the endpoints. Proxy ARP is aware of the location of the traffic destination, and offers its own MAC address as the final destination instead. To enable Proxy ARP, intra-EPG endpoint isolation must be enabled on the EPG see the following figure for details. For more information about intra-EPG isolation and Cisco ACI, see the Cisco ACI Virtualization Guide. Figure 6: Proxy ARP and Cisco APIC Proxy ARP within the Cisco ACI fabric is different from the traditional proxy ARP. As an example of the communication process, when proxy ARP is enabled on an EPG, if an endpoint A sends an ARP request for endpoint B and if endpoint B is learned within the fabric, then endpoint A will receive a proxy ARP response from the bridge domain (BD) MAC. If endpoint A sends an ARP request for endpoint B, and if endpoint B is not learned within the ACI fabric already, then the fabric will send a proxy ARP request within the BD. Endpoint B will respond to this proxy ARP request back to the fabric. At this point, the fabric does not send a proxy ARP response to endpoint A, but endpoint B is learned within the fabric. If endpoint A sends another ARP request to endpoint B, then the fabric will send a proxy ARP response from the BD MAC. The following example describes the proxy ARP resolution steps for communication between clients VM1 and VM2: Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 116 Configuring Layer 2 External Connectivity About Proxy ARP 1 VM1 to VM2 communication is desired. Figure 7: VM1 to VM2 Communication is Desired. Table 9: ARP Table State Device State VM1 IP = * MAC = * ACI fabric IP = * MAC = * VM2 IP = * MAC = * Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 117 Configuring Layer 2 External Connectivity About Proxy ARP 2 VM1 sends an ARP request with a broadcast MAC address to VM2. Figure 8: VM1 sends an ARP Request with a Broadcast MAC address to VM2 Table 10: ARP Table State Device State VM1 IP = VM2 IP; MAC = ? ACI fabric IP = VM1 IP; MAC = VM1 MAC VM2 IP = * MAC = * Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 118 Configuring Layer 2 External Connectivity About Proxy ARP 3 The ACI fabric floods the proxy ARP request within the bridge domain (BD). Figure 9: ACI Fabric Floods the Proxy ARP Request within the BD Table 11: ARP Table State Device State VM1 IP = VM2 IP; MAC = ? ACI fabric IP = VM1 IP; MAC = VM1 MAC VM2 IP = VM1 IP; MAC = BD MAC Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 119 Configuring Layer 2 External Connectivity About Proxy ARP 4 VM2 sends an ARP response to the ACI fabric. Figure 10: VM2 Sends an ARP Response to the ACI Fabric Table 12: ARP Table State Device State VM1 IP = VM2 IP; MAC = ? ACI fabric IP = VM1 IP; MAC = VM1 MAC VM2 IP = VM1 IP; MAC = BD MAC Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 120 Configuring Layer 2 External Connectivity About Proxy ARP 5 VM2 is learned. Figure 11: VM2 is Learned Table 13: ARP Table State Device State VM1 IP = VM2 IP; MAC = ? ACI fabric IP = VM1 IP; MAC = VM1 MAC IP = VM2 IP; MAC = VM2 MAC VM2 IP = VM1 IP; MAC = BD MAC Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 121 Configuring Layer 2 External Connectivity About Proxy ARP 6 VM1 sends an ARP request with a broadcast MAC address to VM2. Figure 12: VM1 Sends an ARP Request with a Broadcast MAC Address to VM2 Table 14: ARP Table State Device State VM1 IP = VM2 IP MAC = ? ACI fabric IP = VM1 IP; MAC = VM1 MAC IP = VM2 IP; MAC = VM2 MAC VM2 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 122 IP = VM1 IP; MAC = BD MAC Configuring Layer 2 External Connectivity Guidelines and Limitations 7 The ACI fabric sends a proxy ARP response to VM1. Figure 13: ACI Fabric Sends a Proxy ARP Response to VM1 Table 15: ARP Table State Device State VM1 IP = VM2 IP; MAC = BD MAC ACI fabric IP = VM1 IP; MAC = VM1 MAC IP = VM2 IP; MAC = VM2 MAC VM2 IP = VM1 IP; MAC = BD MAC Guidelines and Limitations Consider these guidelines and limitations when using Proxy ARP: • Proxy ARP is supported only on isolated EPGs. If an EPG is not isolated, a fault will be raised. For communication to happen within isolated EPGs with proxy ARP enabled, you must configure uSeg EPGs. For example, within the isolated EPG, there could be multiple VMs with different IP addresses, and you can configure a uSeg EPG with IP attributes matching the IP address range of these VMs. • ARP requests from isolated endpoints to regular endpoints and from regular endpoints to isolated endpoints do not use proxy ARP. In such cases, endpoints communicate using the real MAC addresses of destination VMs. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 123 Configuring Layer 2 External Connectivity Configuring Proxy ARP Using the Cisco NX-OS Style CLI Configuring Proxy ARP Using the Cisco NX-OS Style CLI Before You Begin • The appropriate tenant, VRF, bridge domain, application profile and EPG must be created. • Intra-EPG isolation must be enabled on the EPG where proxy ARP has to be enabled. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Enters the tenant configuration mode. Example: apic1(config)# tenant Tenant1 Step 3 application application-profile-name Creates an application profile and enters the application mode. Example: apic1(config-tenant)# application Tenant1-App Step 4 epg application-profile-EPG-name Creates an EPG and enter the EPG mode. Example: apic1(config-tenant-app)# epg Tenant1-epg1 Step 5 proxy-arp enable Enables proxy ARP. Note Example: apic1(config-tenant-app-epg)# proxy-arp enable Step 6 exit You can disable proxy-arp with the no proxy-arp command. Returns to application profile mode. Example: apic1(config-tenant-app-epg)# exit Step 7 exit Example: apic1(config-tenant-app)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 124 Returns to tenant configuration mode. Configuring Layer 2 External Connectivity Configuring Proxy ARP Using the Cisco NX-OS Style CLI Step 8 Command or Action Purpose exit Returns to global configuration mode. Example: apic1(config-tenant)# exit Examples This example shows how to configure proxy ARP. apic1# conf t apic1(config)# tenant Tenant1 apic1(config-tenant)# application Tenant1-App apic1(config-tenant-app)# epg Tenant1-epg1 apic1(config-tenant-app-epg)# proxy-arp enable apic1(config-tenant-app-epg)# apic1(config-tenant)# Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 125 Configuring Layer 2 External Connectivity Configuring Proxy ARP Using the Cisco NX-OS Style CLI Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 126 CHAPTER 6 Configuring Layer 3 External Connectivity • About the Modes of Configuring Layer 3 External Connectivity, page 127 • Configuring Layer 3 External Connectivity, page 129 • Layer 3 Out to Layer 3 Out Inter-VRF Leaking, page 129 • About SVI External Encapsulation Scope, page 133 • About SVI Auto State , page 136 • Configuring an Interface and Static Route , page 138 • OSPF Configuration, page 141 • BGP Configuration, page 148 • EIGRP Configuration, page 164 • Configuring Route-Maps, page 171 • Configuring Bi-Directional Route Forwarding (BFD), page 182 • Configuring Layer 3 Multicast, page 193 • Configuring External-L3 EPGs, page 207 • Configuring Layer 3 External Connectivity Using the Named Mode, page 209 • Configuring HSRP, page 223 • Cisco ACI GOLF, page 226 • Multipod_Fabric, page 241 • Cisco APIC Quality of Service, page 247 About the Modes of Configuring Layer 3 External Connectivity Because APIC supports multiple user interfaces (UIs) for configuration, the potential exists for unintended interactions when you create a configuration with one UI and later modify the configuration with another UI. This section describes considerations for configuring Layer 3 external connectivity with the APIC NX-OS style CLI, when you may also be using other APIC user interfaces. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 127 Configuring Layer 3 External Connectivity About the Modes of Configuring Layer 3 External Connectivity When you configure Layer 3 external connectivity with the APIC NX-OS style CLI, you have the choice of two modes: • Basic (or Implicit) Mode is compatible with the Basic GUI but not the Advanced GUI or API. • Named (or Explicit) Mode is compatible with the Advanced GUI and API but not the Basic GUI. In either case, the configuration should be considered read-only in the incompatible UI. How the Modes Differ In both modes, the configuration settings are defined within an internal container object, the "L3 Outside" (or "L3Out"), which is an instance of the l3extOut class in the API. The main difference between the two modes is in the naming of this container object instance: • Basic—In the Basic Mode, the naming of the container is implicit and does not appear in the CLI commands. The CLI creates and maintains these objects internally. • Named—In the Named Mode, the naming is provided by the user. CLI commands in the Named Mode have an additional l3Out field. To make the correct use of the named L3Out configuration and avoid faults, the user is expected to understand the API object model for external Layer 3 configuration. Note Except for the procedures in the Configuring Layer 3 External Connectivity Using the Named Mode section, this chapter describes Basic Mode procedures. Guidelines and Restrictions • In the same APIC, both modes can be used together for configuring Layer 3 external connectivity with the following restriction: The Layer 3 external connectivity configuration for a given combination of tenant, VRF, and leaf can be done only through one of the two modes. • For a given tenant VRF, the policy domain where the External-l3 EPG can be placed can be in either the Named Mode or in the Basic Mode. The recommended configuration method is to use only one mode for a given tenant VRF combination across all the nodes where the given tenant VRF is deployed for Layer 3 external connectivity. The modes can be different across different tenants or different VRFs and no restrictions apply. • The external Layer 3 features are supported in both configuration modes, with the following exception: ◦Route-peering and Route Health Injection (RHI) with a L4-L7 Service Appliance is supported only in the Named Mode. The Named Mode should be used across all border leaf switches for the tenant VRF where route-peering is involved. • Layer 3 external network objects (l3extOut) created implicitly by Basic Mode CLI procedures are identified by names starting with “__ui_” and are marked as read-only in the Advanced GUI. The CLI partitions these external-l3 networks by function, such as interfaces, protocols, route-map, and EPG. Configuration modifications performed through the API can break this structure, preventing further modification through the CLI. For the steps to remove such objects, see Troubleshooting Unwanted _ui_ Objects in the APIC Troubleshooting Guide. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 128 Configuring Layer 3 External Connectivity Configuring Layer 3 External Connectivity Configuring Layer 3 External Connectivity Configuration of layer 3 (L3) routing connectivity to an external network consists of the following components: • Interface—Interface configuration for layer 3 ports, sub-interfaces, external SVI that are used to connect to external routers. • Routing Protocol Configuration—CLI supports static route, BGP, OSPF, EIGRP protocol configuration. • Route-map control—A route map is used to match prefixes/BD public subnets and apply route-control policies. Once created, it can be associated with routing protocols in a direction, such as “in” (BGP or OSPF), “out”(BGP, OSPF, EIGRP). Configurations pertaining to interface, routing protocols, and route-maps are maintained per leaf switch under the config-leaf configuration mode. • External-L3 EPG—A list of external subnets on a tenant VRF that are classified as one endpoint group for applying contract and QoS policies. External-L3 EPGs (also called prefix EPGs) can have contracts with other external-L3 EPGs and application EPGs. External-L3 EPG configuration is maintained under tenant configuration. The external-L3 EPGs can be deployed on a subset of nodes where the VRF is configured. The steps for configuring layer 3 external connectivity can be summarized as follows: 1 Create a VRF under a tenant. 2 Configure and deploy the VRF on the border leaf switch. 3 Configure layer 3 interfaces on the border leaf Interfaces. 4 Configure route-maps on the leaf switch. 5 Configure routing protocols (BGP, OSPF, EIGRP) under leaf and leaf-interface. 6 Create and configure an external-L3 EPG under a tenant. 7 Deploy the external-L3 EPG on the border leaf switch. Layer 3 Out to Layer 3 Out Inter-VRF Leaking Starting with Cisco APIC release 2.2(2e) , when there are two Layer 3 Outs in two different VRFs, inter-VRF leaking is supported. For this feature to work, the following conditions must be satisfied: • A contract between the two Layer 3 Outs is required. • Routes of connected and transit subnets for a Layer 3 Out are leaked by enforcing contracts (L3Out-L3Out as well as L3Out-EPG) and without leaking the dynamic or static routes between VRFs. • Dynamic or static routes are leaked for a Layer 3 Out by enforcing contracts (L3Out-L3Out as well as L3Out-EPG) and without advertising directly connected or transit routes between VRFs. • Shared Layer 3 Outs in different VRFs can communicate with each other. • Two Layer 3 Outs can be in two different VRFs, and they can successfully exchange routes. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 129 Configuring Layer 3 External Connectivity Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Named Example • This enhancement is similar to the Application EPG to Layer 3 Out inter-VRF communications. The only difference is that instead of an Application EPG there is another Layer 3 Out. Therefore, in this case, the contract is between two Layer 3 Outs. In the following figure, there are two Layer 3 Outs with a shared subnet. There is a contract between the Layer 3 external instance profile (l3extInstP) in both the VRFs. In this case, the Shared Layer 3 Out for VRF1 can communicate with the Shared Layer 3 Out for VRF2. Figure 14: Shared Layer 3 Outs Communicating Between Two VRFs Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Named Example Procedure Command or Action Step 1 Enter the configure mode. Example: apic1# configure Step 2 Configure the provider Layer 3 Out. Example: apic1(config)# tenant t1_provider apic1(config-tenant)# external-l3 epg l3extInstP-1 l3out T0-o1-L3OUT-1 apic1(config-tenant-l3ext-epg)# vrf member VRF1 apic1(config-tenant-l3ext-epg)# match ip 192.168.2.0/24 shared apic1(config-tenant-l3ext-epg)# contract provider vzBrCP-1 apic1(config-tenant-l3ext-epg)# exit apic1(config-tenant)# exit apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant t1_provider vrf VRF1 l3out Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 130 Purpose Configuring Layer 3 External Connectivity Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Implicit Example Command or Action Purpose T0-o1-L3OUT-1 apic1(config-leaf-vrf)# route-map T0-o1-L3OUT-1_shared apic1(config-leaf-vrf-route-map)# ip prefix-list l3extInstP-1 permit 192.168.2.0/24 apic1(config-leaf-vrf-route-map)# match prefix-list l3extInstP-1 apic1(config-leaf-vrf-route-map-match)# exit apic1(config-leaf-vrf-route-map)# exit apic1(config-leaf-vrf)# exit apic1(config-leaf)# exit Step 3 Configure the consumer Layer 3 Out. Example: apic1(config)# tenant t1_consumer apic1(config-tenant)# external-l3 epg l3extInstP-2 l3out T0-o1-L3OUT-1 apic1(config-tenant-l3ext-epg)# vrf member VRF2 apic1(config-tenant-l3ext-epg)# match ip 199.16.2.0/24 shared apic1(config-tenant-l3ext-epg)# contract consumer vzBrCP-1 imported apic1(config-tenant-l3ext-epg)# exit apic1(config-tenant)# exit apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant t1_consumer vrf VRF2 l3out T0-o1-L3OUT-1 apic1(config-leaf-vrf)# route-map T0-o1-L3OUT-1_shared apic1(config-leaf-vrf-route-map)# ip prefix-list l3extInstP-2 permit 199.16.2.0/24 apic1(config-leaf-vrf-route-map)# match prefix-list l3extInstP-2 apic1(config-leaf-vrf-route-map-match)# exit apic1(config-leaf-vrf-route-map)# exit apic1(config-leaf-vrf)# exit apic1(config-leaf)# exit apic1(config)# Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Implicit Example Procedure Command or Action Step 1 Purpose Enter the configure mode. Example: apic1# configure Step 2 Configure the provider tenant and VRF. Example: apic1(config)# tenant t1_provider apic1(config-tenant)# vrf context VRF1 apic1(config-tenant-vrf)# exit apic1(config-tenant)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 131 Configuring Layer 3 External Connectivity Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Implicit Example Command or Action Step 3 Configure the consumer tenant and VRF. Example: apic1(config)# tenant t1_consumer apic1(config-tenant)# vrf context VRF2 apic1(config-tenant-vrf)# exit apic1(config-tenant)# exit Step 4 Configure the contract. Example: apic1(config)# tenant t1_provider apic1(config-tenant)# contract vzBrCP-1 type permit apic1(config-tenant-contract)# scope exportable apic1(config-tenant-contract)# export to tenant t1_consumer apic1(config-tenant-contract)# exit Step 5 Configure the provider External Layer 3 EPG. Example: apic1(config-tenant)# external-l3 epg l3extInstP-1 apic1(config-tenant-l3ext-epg)# vrf member VRF1 apic1(config-tenant-l3ext-epg)# match ip 192.168.2.0/24 shared apic1(config-tenant-l3ext-epg)# contract provider vzBrCP-1 apic1(config-tenant-l3ext-epg)# exit apic1(config-tenant)# exit Step 6 Configure the provider export map. Example: apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant t1_provider vrf VRF1 apic1(config-leaf-vrf)# route-map map1 apic1(config-leaf-vrf-route-map)# ip prefix-list p1 permit 192.168.2.0/24 apic1(config-leaf-vrf-route-map)# match prefix-list p1 apic1(config-leaf-vrf-route-map-match)# exit apic1(config-leaf-vrf-route-map)# exit apic1(config-leaf-vrf)# export map map1 apic1(config-leaf-vrf)# exit apic1(config-leaf)# exit Step 7 Configure the consumer external Layer 3 EPG. Example: apic1(config)# tenant t1_consumer apic1(config-tenant)# external-l3 epg l3extInstP-2 apic1(config-tenant-l3ext-epg)# vrf member VRF2 apic1(config-tenant-l3ext-epg)# match ip 199.16.2.0/24 shared apic1(config-tenant-l3ext-epg)# contract consumer vzBrCP-1 imported apic1(config-tenant-l3ext-epg)# exit apic1(config-tenant)# exit Step 8 Configure the consumer export map. Example: apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant t1_consumer vrf VRF2 apic1(config-leaf-vrf)# route-map map2 apic1(config-leaf-vrf-route-map)# ip prefix-list p2 permit 199.16.2.0/24 apic1(config-leaf-vrf-route-map)# match prefix-list p2 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 132 Purpose Configuring Layer 3 External Connectivity About SVI External Encapsulation Scope Command or Action Purpose apic1(config-leaf-vrf-route-map-match)# exit apic1(config-leaf-vrf-route-map)# exit apic1(config-leaf-vrf)# export map map2 apic1(config-leaf-vrf)# exit apic1(config-leaf)# exit apic1(config)# About SVI External Encapsulation Scope In the context of a Layer 3 Out configuration, a switch virtual interfaces (SVI), is configured to provide connectivity between the ACI leaf switch and a router. By default, when a single Layer 3 Out is configured with SVI interfaces, the VLAN encapsulation spans multiple nodes within the fabric. This happens because the ACI fabric configures the same bridge domain (VXLAN VNI) across all the nodes in the fabric where the Layer 3 Out SVI is deployed as long as all SVI interfaces use the same external encapsulation (SVI) as shown in the figure. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 133 Configuring Layer 3 External Connectivity About SVI External Encapsulation Scope However, when different Layer 3 Outs are deployed, the ACI fabric uses different bridge domains even if they use the same external encapsulation (SVI) as shown in the figure: Figure 15: Local Scope Encapsulation and One Layer 3 Out Figure 16: Local Scope Encapsulation and Two Layer 3 Outs Starting with Cisco APIC release 2.3, it is now possible to choose the behavior when deploying two (or more) Layer 3 Outs using the same external encapsulation (SVI). The encapsulation scope can now be configured as Local or VRF: • Local scope (default): The example behavior is displayed in the figure titled Local Scope Encapsulation and Two Layer 3 Outs. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 134 Configuring Layer 3 External Connectivity Encapsulation Scope Syntax • VRF scope: The ACI fabric configures the same bridge domain (VXLAN VNI) across all the nodes and Layer 3 Out where the same external encapsulation (SVI) is deployed. See the example in the figure titled VRF Scope Encapsulation and Two Layer 3 Outs. Figure 17: VRF Scope Encapsulation and Two Layer 3 Outs Encapsulation Scope Syntax The options for configuring the scope of the encapsulation used for the Layer 3 Out profile are as follows: • Ctx—The same external SVI in all Layer 3 Outs in the same VRF for a given VLAN encapsulation. This is a global value. • Local —A unique external SVI per Layer 3 Out. This is the default value. The mapping among the CLI, API, and GUI syntax is as follows: Table 16: Encapsulation Scope Syntax Note CLI API GUI l3out local Local vrf ctx VRF The CLI commands to configure encapsulation scope are only supported when the VRF is configured through a named Layer 3 Out configuration. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 135 Configuring Layer 3 External Connectivity Configuring SVI Interface Encapsulation Scope Using NX-OS Style CLI Configuring SVI Interface Encapsulation Scope Using NX-OS Style CLI The following example displaying steps for an SVI interface encapsulation scope setting is through a named Layer 3 Out configuration. Procedure Step 1 Command or Action Purpose Enter the configure mode. Enters the configuration mode. Example: apic1# configure Step 2 Enter the switch mode. Enters the switch mode. Example: apic1(config)# leaf 104 Step 3 Create the VLAN interface. Creates the VLAN interface. The VLAN range is 1-4094. Example: apic1(config-leaf)# interface vlan 2001 Step 4 Specify the encapsulation scope. Specifies the encapsulation scope. Example: apic1(config-leaf-if)# encap scope vrf context Step 5 Exit the interface mode. Exits the interface mode. Example: apic1(config-leaf-if)# exit About SVI Auto State Note This APIC Release 2.2(3x) feature is only available in this specific release and no other release. The Switch Virtual Interface (SVI) represents a logical interface between the bridging function and the routing function of a VLAN in the device. SVI can have members that are physical ports, direct port channels, or virtual port channels. The SVI logical interface is associated with VLANs, and the VLANs have port membership. The SVI state does not depend on the members. The default auto state behavior for SVI in Cisco APIC is that it remains in the up state when the auto state value is disabled. This means that the SVI remains active even if no interfaces are operational in the corresponding VLAN/s. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 136 Configuring Layer 3 External Connectivity Guidelines and Limitations for SVI Auto State Behavior If the SVI auto state value is changed to enabled, then it depends on the port members in the associated VLANs. When a VLAN interface has multiple ports in the VLAN, the SVI goes to the down state when all the ports in the VLAN go down. Table 17: SVI Auto State SVI Auto State Description of SVI State Disabled SVI remains in the up state even if no interfaces are operational in the corresponding VLAN/s. Disabled is the default SVI auto state value. SVI depends on the port members in the associated VLANs. When a VLAN interface contains multiple ports, the SVI goes into the down state when all the ports in the VLAN go down. Enabled Guidelines and Limitations for SVI Auto State Behavior Read the following guidelines: • When you enable or disable the auto state behavior for SVI, you configure the auto state behavior per SVI. There is no global command. Configuring SVI Auto State Using NX-OS Style CLI Before You Begin • The tenant and VRF configured. • A Layer 3 Out is configured and a logical node profile and a logical interface profile under the Layer 3 Out is configured. Procedure Step 1 Command or Action Purpose Enter the configure mode. Enters the configuration mode. Example: apic1# configure Step 2 Enter the switch mode. Enters the switch mode. Example: apic1(config)# leaf 104 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 137 Configuring Layer 3 External Connectivity Configuring an Interface and Static Route Step 3 Command or Action Purpose Create the VLAN interface. Creates the VLAN interface. The VLAN range is 1-4094. Example: apic1(config-leaf)# interface vlan 2001 Step 4 Enable SVI auto state. Enables SVI auto state. Example: By default, the SVI auto state value is not enabled. apic1(config-leaf-if)# autostate Step 5 Exit the interface mode. Exits the interface mode. Example: apic1(config-leaf-if)# exit Configuring an Interface and Static Route Before You Begin Configure a tenant and VRF. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 [no] vrf context tenant tenant-name vrf vrf-name Configures a tenant VRF on the node. Example: apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 Step 4 [no] router-id ipv4-address Example: apic1(config-leaf-vrf)# router-id 1.2.3.4 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 138 (Optional) Assigns a router ID for routing protocols running on the VRF. If you do not assign a router ID, an ID is generated internally that is unique to each leaf switch. Configuring Layer 3 External Connectivity Configuring an Interface and Static Route Step 5 Command or Action Purpose [no] {ip | ipv6} route ip-prefix/masklen next-hop-address [preferred] Configures static route information for the VRF. Example: apic1(config-leaf-vrf)# ip route 21.1.1.1/32 32.1.1.1 apic1(config-leaf-vrf)# ipv6 route 5001::1/128 6002::1 Step 6 Returns to leaf configuration mode. exit Example: apic1(config-leaf-vrf)# exit Step 7 interface type Specifies a port for the external interface. Example: apic1(config-leaf)# interface eth 1/1 Step 8 vlan-domain member domain-name Example: apic1(config-leaf-if)# vlan-domain member dom1 Step 9 no switchport Example: apic1(config-leaf-if)# no switchport Step 10 vrf member tenant tenant-name vrf vrf-name Assign a VLAN domain to the interface. The VLAN domain must have already been created using the vlan-domain command in the global configuration mode. Configures the interface as a layer 3 interface, exposing the layer 3 commands in the configuration options. Attaches the interface to the tenant VRF. Example: apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 Step 11 [no] {ip | ipv6} address ip-prefix/masklen [eui64] [secondary] [preferred] Example: apic1(config-leaf-if)# ip address 10.1.1.1/24 apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred Configures IP addresses on the interface. The specified address can be declared as either: • preferred—The default source address for traffic from the interface. • secondary—The secondary address of the interface. With the optional eui64 keyword, the host can assign itself a 64-bit Extended Unique Identifier (EUI). In this mode, you can also configure ipv6 link-local, mac Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 139 Configuring Layer 3 External Connectivity Configuring an Interface and Static Route Command or Action Purpose address, mtu, and other layer 3 properties on the interface. Sets or removes a DHCP relay Step 12 [[no]]ip dhcprelayaddress tenanttenant-name address for the external interface dhcp-address{applicationapp-name epgepg-name|external-1212-epg-name|external-1313-epg-name} along with any supported options. Example: apic(config-leaf-if)# ip dhcp relay address 192.0.20.1 tenant exampleCorp application app1 epg epg1 Examples This example shows how to deploy a layer 3 port for external connectivity. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# vrf apic1(config-leaf-vrf)# apic1(config-leaf-vrf)# apic1(config-leaf-vrf)# apic1(config-leaf-vrf)# context tenant exampleCorp vrf v1 router-id 1.2.3.4 ip route 21.1.1.1/32 32.1.1.1 ipv6 route 5001::1/128 6002::1 preferred exit apic1(config-leaf)# interface eth 1/1 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# no switchport apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 apic1(config-leaf-if)# ip address 10.1.1.1/24 apic1(config-leaf-if)# ip address 11.1.1.1/24 secondary apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred apic1(config-leaf-if)# ipv6 link-local fe80::1 apic1(config-leaf-if)# mac-address 00:44:55:66:55::01 apic1(config-leaf-if)# mtu 4470 This example shows how to configure a layer 3 subinterface port for external connectivity. In this example, the subinterface ID (the "100" in 1/2.100) is actually the VLAN encapsulation instead of an ID. All properties supported on a layer 3 port are available on the subinterface as well. apic1# configure apic1(config)# leaf 101 # SAME VRF CONTEXT CONFIGURATION AS PREVIOUS EXAMPLE apic1(config-leaf)# interface eth 1/2.100 apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 # SAME L3 PROPERTIES CONFIGURATION AS PREVIOUS EXAMPLE This example shows the methods to configure a switched virtual interface (SVI) for external connectivity. Each external SVI is uniquely identified by its encap VLAN denoted in the SVI ID. apic1# configure apic1(config)# leaf 101 # SAME VRF CONTEXT CONFIGURATION AS PREVIOUS EXAMPLE apic1(config-leaf)# interface vlan 200 apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 apic1(config-leaf-if)# ip address 13.1.1.1/24 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 140 Configuring Layer 3 External Connectivity OSPF Configuration # HOW TO ATTACH A PORT TO THE EXTERNAL SVI: apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/4 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf)# switchport trunk allowed vlan 10 tenant exampleCorp external-svi # HOW TO ATTACH A PORT CHANNEL TO THE EXTERNAL SVI: apic1(config)# leaf 102 apic1(config-leaf)# interface port-channel po1 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf)# switchport trunk allowed vlan 10 tenant exampleCorp external-svi # HOW TO ATTACH A VIRTUAL PORT CHANNEL (vPC) TO THE EXTERNAL SVI: apic1(config)# vpc context leaf 101 102 apic1(config-leaf)# interface vpc vpc103 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf)# switchport trunk allowed vlan 10 tenant exampleCorp external-svi Note An external SVI must be configured on each of the participating nodes. This allows you to configure different IP addresses on each of the nodes for the same SVI. If the vPC is part of an external SVI, you must individually create an SVI on each of the participating vPC peers and you can provide different IP addresses on each SVI. OSPF Configuration Configuring OSPF OSPF can operate in one of the following modes in an area: • When OSPF is used as the main routing protocol for the tenant VRF in the node, OSPF will import and export routes defined in the route-map configured in the OSPF area. The route-map contains the export routes. • When OSPF is used as a connectivity protocol for BGP, OSPF advertises the loopback address which is used as the source of the BGP session. Note that the loopback IP address and not the loopback ID is used. In this case, a BGP session relying on OSPF will use the same loopback IP address in its update-source command. There is no need for separate configuration of OSPF and OSPFv3. The router OSPF mode handles both OSPFv2 and OSPFv3 implicitly for the areas running under OSPF. OSPF sessions are supported on all types of layer 3 Interfaces in the leaf, including: • Layer 3 ports • Subinterfaces • External SVI Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 141 Configuring Layer 3 External Connectivity Configuring OSPF Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 router ospf default Creates an OSPF routing process and enters OSPF policy configuration. Example: apic1(config-leaf)# router ospf default Step 4 vrf member tenant tenant-name vrf vrf-name Enables a VRF in the OSPF session. Example: apic1(config-leaf-ospf)# vrf member tenant exampleCorp vrf v100 Step 5 default-information originate [always] (Optional) Causes the switch to generate a default route. Example: apic1(config-leaf-ospf-vrf)# default-information originate Step 6 area area-id nssa [no-redistribution] [default-information-originate] Defines a not-so-stubby area (NSSA). Example: apic1(config-leaf-ospf-vrf)# area 0 nssa Step 7 area area-id stub Defines an area to be a stub area. Example: apic1(config-leaf-ospf-vrf)# area 17 stub Step 8 area area-id default-cost cost Sets OSPF default area cost to a value between 0 and 16777215. Example: apic1(config-leaf-ospf-vrf)# area 17 default-cost 20 Step 9 area area-id route-map map-name out Example: apic1(config-leaf-ospf-vrf)# area 17 route-map ospf-to-eigrp out Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 142 Specifies a route-map for outbound filtering. Configuring Layer 3 External Connectivity Configuring OSPF Step 10 Command or Action Purpose area area-id loopback loopback-address apic1(config-leaf-ospf-vrf)# area 17 loopback 192.0.20.11/32 When OSPF is used as a connectivity protocol for BGP, OSPF advertises the loopback address which is used as the source of the BGP session. Note that the loopback IP address and not the loopback ID is used. In this case, a BGP session relying on OSPF will use the same loopback IP address in its update-source command. inherit {ipv4 | ipv6} ospf vrf-policy policy-name Inherits the OSPF Template Policy under this VRF. Example: Step 11 Example: apic1(config-leaf-ospf-vrf)# inherit ipv4 ospf vrf-policy vrfTemplate2 Step 12 summary-address ip-address Example: Configures external route summarization. Enter the summary address for external routes learned from other protocols. apic1(config-leaf-ospf-vrf)# summary-address 182.1.20.0/24 Step 13 area area-id range address-range cost cost Example: Step 14 Configures inter-area route summarization, which summarizes the networks between areas. apic1(config-leaf-ospf-vrf)# area 17 range 192.0.20.0/24 cost 20 The range is the summary route to be advertised in areas. The cost is a value between 0 and 16777215. exit Returns to OSPF configuration mode. Example: apic1(config-leaf-ospf-vrf)# exit Step 15 Returns to leaf configuration mode. exit Example: apic1(config-leaf-ospf)# exit Step 16 interface slot/port Specifies a port for the OSPF interface. The interface could also be specified as interface slot/port.vlan-id or interface vlanvlan-id. Example: apic1(config-leaf)# interface eth 1/2 Step 17 {ip | ipv6} router ospf default area area-id Creates an OSPF routing process and enters OSPF policy configuration. Example: apic1(config-leaf-if)# ip router ospf default area 17 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 143 Configuring Layer 3 External Connectivity Configuring OSPF Step 18 Command or Action Purpose {ip | ipv6} ospf inherit interface-policy if-policy-name tenant tenant-name Inherits the OSPF interface template policy under this tenant. Example: apic1(config-leaf-if)# ip ospf inherit interface-policy ifPolicy3 tenant exampleCorp Step 19 [no] {ip | ipv6} ospf prefix-suppression {enable | disable | inherit} Example: Prevents OSPF from advertising all IP prefixes that belong to a specific interface, except for prefixes that are associated with secondary IP addresses. apic1(config-leaf-if)# ip ospf prefix-suppression enable Step 20 [no] {ip | ipv6} ospf passive-interface Suppresses routing updates on the interface. Example: apic1(config-leaf-if)# ip ospf passive-interface Step 21 [no] ip ospf authentication {md5 | none | simple} Specifies the authentication type. Example: apic1(config-leaf-if)# ip ospf authentication md5 Step 22 ip ospf authentication-key key Specifies the authentication key. Example: apic1(config-leaf-if)# ip ospf authentication-key c1$c0123 Examples apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router ospf default apic1(config-leaf-ospf)# vrf member tenant exampleCorp vrf v100 apic1(config-leaf-ospf-vrf)# area 0 nssa apic1(config-leaf-ospf-vrf)# area 17 stub apic1(config-leaf-ospf-vrf)# area 17 default-cost 20 apic1(config-leaf-ospf-vrf)# area 17 route-map ospf-to-eigrp out apic1(config-leaf-ospf-vrf)# area 17 loopback 192.0.20.11/32 apic1(config-leaf-ospf-vrf)# inherit ipv4 ospf vrf-policy vrfTemplate2 apic1(config-leaf-ospf-vrf)# summary-address 182.1.20.0/24 apic1(config-leaf-ospf-vrf)# area 17 range 192.0.20.0/24 cost 20 apic1(config-leaf-ospf-vrf)# exit apic1(config-leaf-ospf)# exit apic1(config-leaf)# interface eth 1/3 apic1(config-leaf-if)# ip router ospf default area 17 apic1(config-leaf-if)# ip ospf inherit interface-policy ifPolicy3 tenant exampleCorp apic1(config-leaf-if)# ip ospf prefix-suppression enable apic1(config-leaf-if)# ip ospf passive-interface apic1(config-leaf-if)# ip ospf authentication md5 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 144 Configuring Layer 3 External Connectivity Creating OSPF VRF and Interface Templates apic1(config-leaf-if)# ip ospf authentication-key c1$c0123 Creating OSPF VRF and Interface Templates Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 template ospf vrf-policy vrf-policy-name tenant tenant-name Creates the OSPF VRF policy template under the specified tenant. Example: apic1(config-leaf)# template ospf vrf-policy vrfTemplate3 tenant exampleCorp Step 4 timers throttle lsa start-time hold-interval max-time Sets the start-interval, hold-interval, and max-interval for link-state advertisements (LSA). Example: apic1(config-vrf-policy)# timers throttle lsa 200 10000 45000 Step 5 timers lsa-group-pacing seconds Sets the interval in which LSAs are grouped and refreshed, checksummed, or aged. Example: apic1(config-vrf-policy)# timers lsa-group-pacing 240 Step 6 timers lsa-arrival milliseconds Sets the minimum interval between the arrival of each LSA. Example: apic1(config-vrf-policy)# timers lsa-arrival 1000 Step 7 timers throttle spf spf-start spf-hold spf-max-wait Sets the SPF init-interval, hold-interval, and max-interval for LSA. Example: apic1(config-vrf-policy)# timers throttle spf 5 1000 90000 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 145 Configuring Layer 3 External Connectivity Creating OSPF VRF and Interface Templates Step 8 Command or Action Purpose auto-cost reference-bandwidth bandwidth Sets OSPF Policy Bandwidth Reference in Mbps. Example: apic1(config-vrf-policy)# auto-cost reference-bandwidth 1000 Step 9 distance distance Sets OSPF Policy Preferred Administrative Distance. Example: apic1(config-vrf-policy)# distance 200 Step 10 maximum-paths max-paths Example: Sets the maximum number of parallel routes that OSPF can install in a routing table. The range is from 1 to 16 routes. apic1(config-vrf-policy)# maximum-paths 8 Step 11 graceful-restart helper-disable Disables the graceful restart helper mode. Example: apic1(config-vrf-policy)# graceful-restart helper-disable Step 12 prefix-suppression Example: apic1(config-vrf-policy)# prefix-suppression Step 13 name-lookup Prevents OSPF from advertising all IP prefixes except prefixes that are associated with loopbacks, secondary IP addresses, and passive interfaces. Configures OSPF to look up DNS names. Example: apic1(config-vrf-policy)# name-lookup Step 14 exit Returns to leaf configuration mode. Example: apic1(config-vrf-policy)# exit Step 15 template ospf interface-policy if-policy-name Creates the OSPF interface policy template under the specified tenant. tenant tenant-name Example: apic1(config-leaf)# template ospf interface-policy ifTemplate5 tenant exampleCorp Step 16 [no] advertise-subnet Example: apic1(config-interface-policy)# advertise-subnet Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 146 Advertises the primary IP address subnet mask instead of /32. Configuring Layer 3 External Connectivity Creating OSPF VRF and Interface Templates Step 17 Command or Action Purpose [no] cost if-cost Sets the OSPF cost for the interface. The range is 0 to 65535. Example: apic1(config-interface-policy)# cost 300 Step 18 [no] dead-interval seconds Example: apic1(config-interface-policy)# dead-interval 60 Step 19 [no] hello-interval seconds Sets the interval in seconds at which hello packets must not be seen before neighbors declare the router down. The range is 1 to 65535 seconds. Specifies the interval between hello packets in seconds. The range is 1 to 65535 seconds. Example: apic1(config-interface-policy)# hello-interval 10 Step 20 Disables MTU mismatch detection on the interface. [no] mtu-ignore Example: apic1(config-interface-policy)# mtu-ignore Step 21 [no] network {bcast | p2p | unspecified} Sets the OSPF interface policy network type, which can be broadcast or point-to-point. Example: apic1(config-interface-policy)# network p2p Step 22 Suppresses OSPF routing updates on the interface. [no] passive-interface Example: apic1(config-interface-policy)# passive-interface Step 23 [no] priority priority Sets OSPF interface priority, which is used to determine the designated router (DR) on a specific network. The range is 0 to 255. Example: apic1(config-interface-policy)# priority 4 Step 24 [no] retransmit-interval seconds Example: apic1(config-interface-policy)# retransmit-interval 5 Step 25 [no] transmit-delay seconds Example: Specifies the time between link-state advertisement (LSA) retransmissions for adjacencies belonging to the interface. The range is 1 to 65535 seconds. Sets the estimated time required to send a link-state update packet on the interface. The range is from 1 to 450 seconds. apic1(config-interface-policy)# transmit-delay 2 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 147 Configuring Layer 3 External Connectivity BGP Configuration Examples This example shows how to configure a VRF template and an interface template. apic1# configure apic1(config)# leaf 101 # CONFIGURING THE VRF TEMPLATE: apic1(config-leaf)# template ospf vrf-policy vrfTemplate3 tenant exampleCorp apic1(config-vrf-policy)# timers throttle lsa 200 10000 45000 apic1(config-vrf-policy)# timers lsa-group-pacing 240 apic1(config-vrf-policy)# timers lsa-arrival 1000 apic1(config-vrf-policy)# timers throttle spf 5 1000 90000 apic1(config-vrf-policy)# auto-cost reference-bandwidth 1000 apic1(config-vrf-policy)# distance 200 apic1(config-vrf-policy)# maximum-paths 8 apic1(config-vrf-policy)# graceful-restart helper-disable apic1(config-vrf-policy)# prefix-suppression apic1(config-vrf-policy)# name-lookup apic1(config-vrf-policy)# exit # CONFIGURING THE INTERFACE TEMPLATE: apic1(config-leaf)# template ospf interface-policy ifTemplate5 tenant exampleCorp apic1(config-ospf-if-policy)# advertise-subnet apic1(config-ospf-if-policy)# cost 300 apic1(config-ospf-if-policy)# dead-interval 60 apic1(config-ospf-if-policy)# hello-interval 10 apic1(config-ospf-if-policy)# mtu-ignore apic1(config-ospf-if-policy)# network p2p apic1(config-ospf-if-policy)# passive-interface apic1(config-ospf-if-policy)# priority 4 apic1(config-ospf-if-policy)# retransmit-interval 5 apic1(config-ospf-if-policy)# transmit-delay 2 BGP Configuration Configuring BGP Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 bgp-fabric Enters BGP configuration mode for the fabric. Example: apic1(config)# bgp-fabric Step 3 asn asn-number Example: apic1(config-bgp-fabric)# asn 100 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 148 Specifies the BGP autonomous system number (ASN). Configuring Layer 3 External Connectivity Creating BGP Address Family and Timer Templates Step 4 Command or Action Purpose route-reflector spine spine-id Configures the specified spine switch to be a BGP route reflector. Example: apic1(config-bgp-fabric)# route-reflector spine 105 Examples apic1# configure apic1(config)# bgp-fabric apic1(config-bgp-fabric)# asn 100 apic1(config-bgp-fabric)# route-reflector spine 105 What to Do Next Configure BGP address family and counters. Creating BGP Address Family and Timer Templates Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 template bgp timers timer-policy-name tenant tenant-name Creates the BGP timers policy template under the specified tenant. Example: apic1(config-leaf)# template bgp timers bgpTimers tenant exampleCorp This template will be available on all leaves where tenant exampleCorp has a VRF deployment Step 4 graceful-restart-helper Configure BGP Policy Graceful Restart Helper apic1(config-bgp-timers)# graceful-restart-helper Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 149 Configuring Layer 3 External Connectivity Creating BGP Address Family and Timer Templates Step 5 Command or Action Purpose graceful-restart stalepath-time seconds Sets the maximum time that BGP keeps stale routes from the restarting BGP peer. The range is 1 to 3600 seconds. apic1(config-bgp-timers)# graceful-restart stalepath-time 3600 Step 6 timers bgp keep-alive-seconds hold-seconds Sets the keep-alive timer and hold timer values. The range for both is 1 to 3600 seconds. apic1(config-bgp-timers)# timers bgp 10 20 apic1(config-bgp-timers)# exit Step 7 exit Step 8 template bgp address-family family-name Creates the BGP address family template under the specified tenant. tenant tenant-name Example: apic1(config-leaf)# template bgp address-family bgpAf1 tenant exampleCorp This template will be available on all leaves where tenant exampleCorp has a VRF deployment Step 9 distance ebgp-distance ibgp-distance local-distance Sets the administrative distance for eBGP routes, iBGP routes, and local routes. The range is 1 to 255. apic1(config-bgp-af)# distance 250 240 230 Step 10 exit Returns to leaf configuration mode. apic1(config-bgp-af)# exit Examples This example shows how to create a BGP timer template and an address family template. apic1# configure apic1(config)# leaf 101 # CREATE A TIMER TEMPLATE apic1(config-leaf)# template bgp timers bgpTimers tenant exampleCorp This template will be available on all leaves where tenant exampleCorp has a VRF deployment apic1(config-bgp-timers)# timers bgp 10 20 apic1(config-bgp-timers)# graceful-restart stalepath-time 3600 apic1(config-bgp-timers)# exit # CREATE AN ADDRESS FAMILY TEMPLATE apic1(config-leaf)# template bgp address-family bgpAf1 tenant bgp_t1 This template will be available on all leaves where tenant exampleCorp has a VRF deployment apic1(config-bgp-af)# distance 250 240 230 apic1(config-bgp-af)# exit apic1(config-leaf)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 150 Configuring Layer 3 External Connectivity Configuring BGP Address Family and Timers Configuring BGP Address Family and Timers Before You Begin Create a BGP address family template and timer template. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 router bgp asn-number Enters BGP policy configuration. Example: apic1(config-leaf)# router bgp 100 Step 4 vrf member tenant tenant-name vrf vrf-name Example: Specifies the VRF instance to associate with subsequent address family configuration mode commands. apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 Step 5 inherit bgp timer timer-name Applies an existing timer configuration. Example: apic1(config-leaf-bgp-vrf)# inherit bgp timer bgpTimers This template will be inherited on all leaves where VRF v100 has been deployed Step 6 address-family {ipv4 | ipv6} unicast Declares neighbors with whom we want to exchange normal IPv4 unicast routes. Example: apic1(config-leaf-bgp-vrf)# address-family ipv4 unicast Step 7 inherit bgp address-family family-name Adds the specified address family to this address family. Example: apic1(config-leaf-bgp-vrf-af)# inherit bgp address-family ipv4-af-pol This template will be inherited on all leaves where VRF v100 has been deployed Step 8 exit Example: apic1(config-leaf-bgp-vrf-af)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 151 Configuring Layer 3 External Connectivity Configuring a BGP Neighbor Examples This example shows how to inherit a BGP timer configuration and IPv4 and IPv6 address families. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router bgp 100 apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 apic1(config-leaf-bgp-vrf)# inherit bgp timer bgpTimers This template will be inherited on all leaves where VRF v100 has been deployed apic1(config-leaf-bgp-vrf)# address-family ipv4 unicast apic1(config-leaf-bgp-vrf-af)# inherit bgp address-family ipv4-af-pol This template will be inherited on all leaves where VRF v100 has been deployed apic1(config-leaf-bgp-vrf-af)# exit apic1(config-leaf-bgp-vrf)# address-family ipv6 unicast apic1(config-leaf-bgp-vrf-af)# inherit bgp address-family ipv6-af-pol This template will be inherited on all leaves where VRF v100 has been deployed apic1(config-leaf-bgp-vrf-af)# exit apic1(config-leaf-bgp-vrf)# exit apic1(config-leaf)# exit Configuring a BGP Neighbor Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 router bgp asn-number Enters BGP policy configuration. Example: apic1(config-leaf)# router bgp 100 Step 4 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with subsequent policy configuration mode commands. Example: apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 152 Configuring Layer 3 External Connectivity Configuring a BGP Neighbor Step 5 Command or Action Purpose aggregate-address ip-address/masklength [as-set] apic1(config-leaf-bgp-vrf)# aggregate-address 192.0.10.0/24 as-set (Optional) Configures a summary address for a range of addresses and creates an aggregate entry in a BGP database. The address can be either IPv4 or IPv6. The as-set option generates autonomous system set path information. neighbor ip-address[ /masklength ] Specifies the IP address of the neighbor. Example: Step 6 Example: apic1(config-leaf-bgp-vrf)# neighbor 192.0.2.229/32 Step 7 address-family {ipv4 | ipv6} unicast Declares neighbors with whom we want to exchange normal IPv4 unicast routes. Example: apic1(config-leaf-bgp-vrf-neighbor)# address-family ipv4 unicast Step 8 [no] maximum-prefix count [action {log | shut | restart [restart-timeminutes]}] [threshold percent] Example: apic1(config-leaf-bgp-vrf-neighbor-af)# maximum-prefix 10 threshold 10 action restart restart-time 10 Sets the maximum number of prefixes from this neighbor. the range is 1 to 300000 prefixes. Other optional settings are: • action— The action to be performed when the maximum prefix limit is reached. If the action is restart, you can optionally specify the restart-time, which is the period of time in minutes before restarting the peer when the maximum prefix limit is reached. The range is 1 to 65535 minutes. • threshold— The threshold percentage of the maximum number of prefixes before a warning is issued. The range is 1 to 100 percent. Step 9 exit Example: apic1(config-leaf-bgp-vrf-neighbor-af)# exit Step 10 update-source {loopback ip-address | ethernet ip-address | vlan vlan-id} if the neighbor address is being learned through OSPF, specify the same loopback address as being used under OSPF. Example: apic1(config-leaf-bgp-vrf-neighbor)# update-source loopback 192.0.2.230 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 153 Configuring Layer 3 External Connectivity Configuring a BGP Neighbor Step 11 Command or Action Purpose weight number Specifies the weight attribute to select a best path. A weight can from 0 to 65,535. Routes with a higher weight value have preference when there are multiple routes to the same destination. Example: apic1(config-leaf-bgp-vrf-neighbor)# weight 2000 Step 12 private-as-control {remove-exclusive | remove-exclusive-all | remove-exclusive-all-replace-as} Example: apic1(config-leaf-bgp-vrf-neighbor)# private-as-control Removes private autonomous system numbers from the autonomous system path. Private AS numbers can be removed from the AS path on a per peer basis and can only be used for eBGP peers according to the following three possible variations: • remove-exclusive—Remove if AS path has only private AS numbers. • remove-exclusive-all—Remove if AS path has both private and public AS numbers. • remove-exclusive-all-replace-as—Replaces private AS numbers with the router’s local AS number. This command is shown as an example. At this point you can configure any of the neighbor settings shown in the table below. The following table shows the interface settings that can be configured at this point. Command Purpose allow-self-as Accept as-path with my AS present in it allowed-self-as-count count The number of occurrences of a local access service network disable-connected-check Disable check for directly connected peer disable-peer-as-check Disable checking of peer AS-number while advertising ebgp-multihop count Specify multihop TTL for remote peer local-as asn Local Autonomous System Configuration for a BGP Peer next-hop-self Set our peering address as nexthop password password Configure a password for neighbor Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 154 Configuring Layer 3 External Connectivity Configuring a BGP Neighbor Command Purpose private-as-control Removes private ASNs from the AS path remote-as asn Specify Autonomous System Number of the neighbor route-map name {in | out} Apply route-map to neighbor send-community [extended] Send Community attribute to this neighbor update-source vlan vlan-id Source Vlan Interface update-source ethernet slot/port Source Ethernet Interface update-source loopback ip-address Source Loopback Interface weight number BGP weight for the routing table Examples This example shows how to configure an IPv4 BGP neighbor. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router bgp 100 apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 apic1(config-leaf-bgp-vrf)# aggregate-address 192.0.10.0/24 as-set apic1(config-leaf-bgp-vrf)# neighbor 192.0.2.229/32 apic1(config-leaf-bgp-vrf-neighbor)# address-family ipv4 unicast apic1(config-leaf-bgp-vrf-neighbor-af)# maximum-prefix 10 threshold 10 action restart restart-time 10 apic1(config-leaf-bgp-vrf-neighbor-af)# exit apic1(config-leaf-bgp-vrf-neighbor)# allow-self-as apic1(config-leaf-bgp-vrf-neighbor)# allowed-self-as-count 2 apic1(config-leaf-bgp-vrf-neighbor)# disable-connected-check apic1(config-leaf-bgp-vrf-neighbor)# disable-peer-as-check apic1(config-leaf-bgp-vrf-neighbor)# ebgp-multihop 4 apic1(config-leaf-bgp-vrf-neighbor)# local-as 100 apic1(config-leaf-bgp-vrf-neighbor)# next-hop-self apic1(config-leaf-bgp-vrf-neighbor)# password abcdef apic1(config-leaf-bgp-vrf-neighbor)# remote-as 200 apic1(config-leaf-bgp-vrf-neighbor)# send-community extended apic1(config-leaf-bgp-vrf-neighbor)# update-source vlan 601 apic1(config-leaf-bgp-vrf-neighbor)# update-source ethernet 1/15 apic1(config-leaf-bgp-vrf-neighbor)# update-source loopback 192.0.2.230 Warning: BGP Configuration changed. Please re-configure BGP Password if it was enabled apic1(config-leaf-bgp-vrf-neighbor)# local-as 100 no-prepend replace-as dual-as apic1(config-leaf-bgp-vrf-neighbor)# route-map rMapT3 out apic1(config-leaf-bgp-vrf-neighbor)# weight 2000 apic1(config-leaf-bgp-vrf-neighbor)# private-as-control apic1(config-leaf-bgp-vrf-neighbor)# exit apic1(config-leaf-bgp-vrf)# exit apic1(config-leaf)# exit This example shows how to configure an IPv6 BGP neighbor. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router bgp 100 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 155 Configuring Layer 3 External Connectivity Configuring a Per VRF Per Node BGP Timer Policy Using the NX-OS Style CLI apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 apic1(config-leaf-bgp-vrf)# neighbor 2001:80:1:2::229 apic1(config-leaf-bgp-vrf-neighbor)# address-family ipv6 unicast apic1(config-leaf-bgp-vrf-neighbor-af)# maximum-prefix 100 apic1(config-leaf-bgp-vrf-neighbor-af)# exit apic1(config-leaf-bgp-vrf-neighbor)# allow-self-as apic1(config-leaf-bgp-vrf-neighbor)# allowed-self-as-count 2 apic1(config-leaf-bgp-vrf-neighbor)# disable-connected-check apic1(config-leaf-bgp-vrf-neighbor)# disable-peer-as-check apic1(config-leaf-bgp-vrf-neighbor)# ebgp-multihop 4 apic1(config-leaf-bgp-vrf-neighbor)# local-as 100 apic1(config-leaf-bgp-vrf-neighbor)# next-hop-self apic1(config-leaf-bgp-vrf-neighbor)# password abcdef apic1(config-leaf-bgp-vrf-neighbor)# remote-as 200 apic1(config-leaf-bgp-vrf-neighbor)# send-community extended apic1(config-leaf-bgp-vrf-neighbor)# update-source vlan 601 apic1(config-leaf-bgp-vrf-neighbor)# update-source ethernet 1/15 apic1(config-leaf-bgp-vrf-neighbor)# update-source loopback 2001:80:1:2::230/128 Warning: BGP Configuration changed. Please re-configure BGP Password if it was enabled apic1(config-leaf-bgp-vrf-neighbor)# local-as 100 no-prepend replace-as dual-as apic1(config-leaf-bgp-vrf-neighbor)# route-map rMapT3 out apic1(config-leaf-bgp-vrf-neighbor)# weight 2000 apic1(config-leaf-bgp-vrf-neighbor)# private-as-control apic1(config-leaf-bgp-vrf-neighbor)# exit apic1(config-leaf-bgp-vrf-af)# exit apic1(config-leaf-bgp-vrf)# exit apic1(config-leaf)# exit Configuring a Per VRF Per Node BGP Timer Policy Using the NX-OS Style CLI Procedure Command or Action Step 1 Purpose Configure BGP ASN and the route reflector before creating a timer policy. Example: apic1(config)# apic1(config)# bgp-fabric apic1(config-bgp-fabric)# route-reflector spine 102 apic1(config-bgp-fabric)# asn 42 apic1(config-bgp-fabric)# exit apic1(config)# exit apic1# Step 2 Create a timer policy. Example: apic1# config apic1(config)# leaf 101 apic1(config-leaf)# template bgp timers pol7 tenant tn1 This template will be available on all nodes where tenant tn1 has a VRF deployment apic1(config-bgp-timers)# timers bgp 120 240 apic1(config-bgp-timers)# graceful-restart stalepath-time 500 apic1(config-bgp-timers)# maxas-limit 300 apic1(config-bgp-timers)# exit apic1(config-leaf)# exit apic1(config)# exit apic1# Step 3 Display the configured BGP policy. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 156 The specific values are provided as examples only. Configuring Layer 3 External Connectivity Configuring BGP Max Path Command or Action Purpose Example: apic1# show run leaf 101 template bgp timers pol7 # Command: show running-config leaf 101 template bgp timers pol7 leaf 101 template bgp timers pol7 tenant tn1 timers bgp 120 240 graceful-restart stalepath-time 500 maxas-limit 300 exit exit Step 4 Refer to a specific policy at a node. Example: apic1# config apic1(config)# leaf 101 apic1(config-leaf)# router bgp 42 apic1(config-leaf-bgp)# vrf member tenant tn1 vrf ctx1 apic1(config-leaf-bgp-vrf)# inherit node-only bgp timer pol7 apic1(config-leaf-bgp-vrf)# exit apic1(config-leaf-bgp)# exit apic1(config-leaf)# exit apic1(config)# exit apic1# Step 5 Display the node specific BGP timer policy. Example: apic1# show run leaf 101 router bgp 42 vrf member tenant tn1 vrf ctx1 # Command: show running-config leaf 101 router bgp 42 vrf member tenant tn1 vrf ctx1 leaf 101 router bgp 42 vrf member tenant tn1 vrf ctx1 inherit node-only bgp timer pol7 exit exit exit apic1# Configuring BGP Max Path Before You Begin The appropriate tenant and the BGP external routed network are created and available. The following feature enables you to add the maximum number of paths to the route table to enable equal cost, multipath load balancing. The two properties which enable you to configure more paths are maxEcmp and maxEcmpIbgp in the bgpCtxAfPol object. After you configure these two properties, they are propagated to the rest of your implementation. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 157 Configuring Layer 3 External Connectivity Configuring AS Path Prepend Use the following commands when logged in to BGP: maximum-paths [ibgp] no maximum-paths [ibgp] Example Configuration: Procedure Example: apic1(config)# leaf 101 apic1(config-leaf)# template bgp address-family newAf tenant t1 This template will be available on all nodes where tenant t1 has a VRF deployment apic1(config-bgp-af)# maximum-paths ? <1-16> Maximum number of equal-cost paths for load sharing. The default is 16. ibgp Configure multipath for IBGP paths apic1(config-bgp-af)# maximum-paths 10 apic1(config-bgp-af)# maximum-paths ibpg 8 apic1(config-bgp-af)# end apic1# no maximum-paths [ibgp] Configuring AS Path Prepend A BGP peer can influence the best-path selection by a remote peer by increasing the length of the AS-Path attribute. AS-Path Prepend provides a mechanism that can be used to increase the length of the AS-Path attribute by prepending a specified number of AS numbers to it. AS-Path prepending can only be applied in the outbound direction using route-maps. AS Path prepending does not work in iBGP sessions. The AS Path Prepend feature enables modification as follows: Prepend Appends the specified AS number to the AS path of the route matched by the route map. Note • You can configure more than one AS number. • 4 byte AS numbers are supported. • You can prepend a total 32 AS numbers. You must specify the order in which the AS Number is inserted into the AS Path attribute. Prepend-last-as Prepends the last AS numbers to the AS path with a range between 1 and 10. The following table describes the selection criteria for implementation of AS Path Prepend: Prepend 1 Prepend the specified AS number. Prepend-last-as 2 Prepend the last AS numbers to the AS path. DEFAULT Prepend(1) Prepend the specified AS number. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 158 Configuring Layer 3 External Connectivity Route Distribution Into BGP Configuring AS Path Prepend Using the NX-OS Style CLI This section provides information on how to configure the AS Path Prepend feature using the NX-OS style command line interface (CLI). Before You Begin A configured tenant. Procedure To modify the autonomous system path (AS Path) for Border Gateway Protocol (BGP) routes, you can use the set as-path command. The set as-path command takes the form of apic1(config-leaf-vrf-template-route-profile)# set as-path {'prepend as-num [ ,... as-num ] | prepend-last-as num} Example: apic1(config)# leaf 103 apic1(config-leaf)# vrf context tenant t1 vrf v1 apic1(config-leaf-vrf)# template route-profile rp1 apic1(config-leaf-vrf-template-route-profile)# set as-path ? prepend Prepend to the AS-Path prepend-last-as Prepend last AS to the as-path apic1(config-leaf-vrf-template-route-profile)# set as-path prepend 100, 101, 102, 103 apic1(config-leaf-vrf-template-route-profile)# set as-path prepend-last-as 8 apic1(config-leaf-vrf-template-route-profile)# exit apic1(config-leaf-vrf)# exit apic1(config-leaf)# exit What to Do Next To disable AS Path prepend, use the no form of the shown command: apic1(config-leaf-vrf-template-route-profile)# [no] set as-path { prepend as-num [ ,... as-num ] | prepend-last-as num} Route Distribution Into BGP Configuring a Route-Profile with Tenant Scope Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 159 Configuring Layer 3 External Connectivity Route Distribution Into BGP Step 3 Command or Action Purpose template route-profile profile-name tenant tenant-name Creates a route-profile template under tenant for BGP dampening and route redistribution. Example: apic1(config-leaf)# template route-profile map_eigrp tenant exampleCorp Step 4 [no] set tag name Sets the tag value. The name parameter is an unsigned integer. Example: apic1(config-leaf-template-route-profile)# set tag 200 Step 5 Returns to leaf configuration mode. exit Example: apic1(config-leaf-template-route-profile)# exit Step 6 template route-profile profile-name tenant tenant-name Creates a route-profile template under tenant for BGP dampening and route redistribution. Example: apic1(config-leaf)# template route-profile map_ospf tenant exampleCorp Step 7 [no] set tag name Sets the tag value. The name parameter is an unsigned integer. Example: apic1(config-leaf-template-route-profile)# set tag 100 apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# template route-profile apic1(config-leaf-template-route-profile)# apic1(config-leaf-template-route-profile)# apic1(config-leaf)# template route-profile apic1(config-leaf-template-route-profile)# apic1(config-leaf-template-route-profile)# map_eigrp tenant exampleCorp set tag 200 exit map_ospf tenant exampleCorp set tag 100 exit What to Do Next Configure a redistribute route-profile under BGP for OSPF and EIGRP using one of the route-profiles created in this procedure. Configuring a Redistribute Route-Profile Before You Begin Create a route-profile template under tenant for route redistribution. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 160 Configuring Layer 3 External Connectivity Configuring BGP Route Dampening Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 router bgp asn-number Enters BGP policy configuration. Example: apic1(config-leaf)# router bgp 100 Step 4 vrf member tenant tenant-name vrf vrf-name Example: Specifies the VRF instance to associate with subsequent policy configuration mode commands. apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 Step 5 redistribute {ospf | eigrp} route-map map-name Example: apic1(config-leaf-bgp-vrf)# redistribute ospf route-map map_ospf This example configures a redistribute route-profile under BGP for OSPF and EIGRP using the route-profiles created in the example in Creating a Route-Profile with Tenant Scope. The redistribute route-map allows (permits) all routes and applies the route-profile for the route-control actions. In this example, all EIGRP learned routes will be redistributed into BGP with tag 200 and OSPF routes will be redistributed into BGP with tag 100. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router bgp 100 apic1(config-bgp)# vrf member tenant exampleCorp vrf v1 apic1(config-leaf-bgp-vrf)# redistribute eigrp route-map map_eigrp apic1(config-leaf-bgp-vrf)# redistribute ospf route-map map_ospf Configuring BGP Route Dampening BGP route dampening minimizes propagation into the fabric of flapping eBGP routes received from external routers connected to border leaf switches (BLs). Frequently flapping routes from external routers are suppressed on BLs based on configured criteria and prohibited from redistribution to iBGP peers (ACI spine switches). Suppressed routes are reused after a configured time criteria. Each flap penalizes the eBGP route with a penalty of 1000. When the flap penalty reaches a defined suppress-limit threshold (default 2000) the eBGP route is marked as dampened. Dampened routes are not advertised to other BGP peers. The penalty is decremented Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 161 Configuring Layer 3 External Connectivity Configuring BGP Route Dampening to half after every half-life interval (the default is 15 minutes). A dampened route is reused if the penalty falls below a specified reuse-limit (the default is 750). A dampened route is suppressed at most for a specified maximum suppress time (maximum of 45 minutes). Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 template route-profile profile-name tenant tenant-name Creates a route-profile template under tenant for BGP dampening and route redistribution. Example: apic1(config-leaf)# template route-profile damp_rp tenant exampleCorp Step 4 [no] set dampening half-life reuse suppress max-suppress-time Example: apic1(config-leaf-template-route-profile)# set dampening 15 750 2000 60 Configures route flap dampening behavior. The parameters are: • half-life—Decay half life, which is the time in minutes after which a penalty is decreased. Once the route has been assigned a penalty, the penalty is decreased by half after the half life period. The range is 1 to 60 minutes; the default is 15 minutes. • reuse—A route is unsuppressed (reused) if the penalty for a flapping route decreases enough to fall below this value. The range is 1 to 20000; the default is 750. • suppress—A route is suppressed when its penalty exceeds this limit. The range is 1 to 20000; the default is 2000. • max-suppress-time—The maximum time in minutes that a stable route can be suppressed. The range is 1 to 255. Step 5 exit Example: apic1(config-leaf-template-route-profile)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 162 Returns to leaf configuration mode. Configuring Layer 3 External Connectivity Configuring BGP Route Dampening Step 6 Command or Action Purpose router bgp asn-number Enters BGP policy configuration. Example: apic1(config-leaf)# router bgp 100 Step 7 vrf member tenant tenant-name vrf vrf-name Example: Specifies the VRF instance to associate with subsequent policy configuration mode commands. apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 Step 8 neighbor ip-address[ /masklength ] Specifies the IP address of the neighbor. The mask length must be 32. Example: apic1(config-leaf-bgp-vrf)# neighbor 192.0.2.229/32 Step 9 address-family {ipv4 | ipv6} unicast Declares neighbors with whom we want to exchange normal IPv4 unicast routes. Example: apic1(config-leaf-bgp-vrf-neighbor)# address-family ipv4 unicast Step 10 inherit bgp dampening profile-name Example: apic1(config-leaf-bgp-vrf-neighbor-af)# inherit bgp dampening damp_rp Step 11 exit Example: apic1(config-leaf-bgp-vrf-neighbor-af)# exit Step 12 exit Example: apic1(config-leaf-bgp-vrf-neighbor)# exit Step 13 address-family {ipv4 | ipv6} unicast Declares neighbors with whom we want to exchange normal IPv4 unicast routes. Example: apic1(config-leaf-bgp-vrf)# address-family ipv4 unicast Step 14 inherit bgp dampening profile-name Example: apic1(config-leaf-bgp-vrf-af)# inherit bgp dampening damp_rp Step 15 exit Example: apic1(config-leaf-bgp-vrf-af)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 163 Configuring Layer 3 External Connectivity EIGRP Configuration apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# template route-profile damp_rp tenant exampleCorp apic1(config-leaf-template-route-profile)# set dampening 15 750 2000 60 apic1(config-leaf-template-route-profile)# exit apic1(config-leaf)# router bgp 100 apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 apic1(config-leaf-bgp-vrf)# neighbor 192.0.2.229/32 apic1(config-leaf-bgp-vrf-neighbor)# address-family ipv4 unicast apic1(config-leaf-bgp-vrf-neighbor-af)# inherit bgp dampening damp_rp apic1(config-leaf-bgp-vrf-neighbor-af)# exit apic1(config-leaf-bgp-vrf)# address-family ipv6 unicast apic1(config-leaf-bgp-vrf-af)# inherit bgp dampening damp_rp apic1(config-leaf-bgp-vrf-af)# exit EIGRP Configuration Creating EIGRP VRF and Interface Templates Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 template eigrp vrf-policy vrf-policy-name tenant tenant-name Creates the EIGRP VRF policy template under the specified tenant. Example: apic1(config-leaf)# template eigrp vrf-policy vrfTemplate3 tenant exampleCorp This template will be available on all leaves where tenant exampleCorp has a VRF deployment Step 4 distance internal external Example: apic1(config-template-eigrp-vrf-pol)# distance 2 5 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 164 Sets EIGRP administrative distance preference for internal and external routes. The distances can be 1 to 255. Configuring Layer 3 External Connectivity Creating EIGRP VRF and Interface Templates Step 5 Command or Action Purpose maximum-paths limit Sets EIGRP Maximum Path Limit for the VRF policy template. The limit can be 1 to 32. Example: apic1(config-template-eigrp-vrf-pol)# maximum-paths 8 Step 6 Sets EIGRP metric style to wide metric (64 bits). metric version 64bit Example: apic1(config-template-eigrp-vrf-pol)# metric version 64bit Step 7 timers active-time minutes Sets EIGRP active timer interval. The range is 1 to 65535 minutes. Example: apic1(config-template-eigrp-vrf-pol)# timers active-time 1 Step 8 template eigrp interface-policy if-policy-name tenant tenant-name Creates the EIGRP interface policy template under the specified tenant. Example: apic1(config-leaf)# template eigrp interface-policy ifTemplate5 tenant exampleCorp This template will be available on all leaves where tenant exampleCorp has a VRF deployment Step 9 ip hello-interval eigrp default seconds Sets EIGRP hello interval time. The range is 1 to 65535 seconds. Example: apic1(config-template-eigrp-if-pol)# ip hello-interval eigrp default 10 Step 10 ip hold-interval eigrp default seconds Sets EIGRP hold interval time. The range is 1 to 65535 seconds. Example: apic1(config-template-eigrp-if-pol)# ip hold-interval eigrp default 10 Step 11 ip next-hop-self eigrp default Sets EIGRP next-hop-self flag. Example: apic1(config-template-eigrp-if-pol)# ip next-hop-self eigrp default Step 12 ip passive-interface eigrp default Set EIGRP passive-interface flag. Example: apic1(config-template-eigrp-if-pol)# ip passive-interface eigrp default Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 165 Configuring Layer 3 External Connectivity Configuring EIGRP Address Family and Counters Step 13 Command or Action Purpose ip split-horizon eigrp default Sets EIGRP split-horizon flag. Example: apic1(config-template-eigrp-if-pol)# ip split-horizon eigrp default Step 14 exit Returns to leaf configuration mode. Example: apic1(config-template-eigrp-if-pol)# exit Examples apic1# configure apic1(config)# leaf 101 # CONFIGURING THE VRF TEMPLATE: apic1(config-leaf)# template eigrp vrf-policy vrfTemplate3 tenant exampleCorp This template will be available on all leaves where tenant exampleCorp has a VRF deployment apic1(config-template-eigrp-vrf-pol)# distance 2 5 apic1(config-template-eigrp-vrf-pol)# maximum-paths 8 apic1(config-template-eigrp-vrf-pol)# metric version 64bit apic1(config-template-eigrp-vrf-pol)# timers active-time 1 apic1(config-template-eigrp-vrf-pol)# exit # CONFIGURING THE INTERFACE TEMPLATE: apic1(config-leaf)# template eigrp interface-policy ifTemplate5 tenant exampleCorp This template will be available on all leaves where tenant exampleCorp has a VRF deployment apic1(config-template-eigrp-if-pol)# ip hello-interval eigrp default 5 apic1(config-template-eigrp-if-pol)# ip hold-interval eigrp default 10 apic1(config-template-eigrp-if-pol)# ip next-hop-self eigrp default apic1(config-template-eigrp-if-pol)# ip passive-interface eigrp default apic1(config-template-eigrp-if-pol)# ip split-horizon eigrp default apic1(config-template-eigrp-if-pol)# exit apic1(config-leaf)# exit apic1(config)# exit What to Do Next Configure EIGRP address family and counters. Configuring EIGRP Address Family and Counters Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 166 Configuring Layer 3 External Connectivity Configuring EIGRP Address Family and Counters Step 2 Command or Action Purpose leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 Enters EIGRP policy configuration. router eigrp default Example: apic1(config-leaf)# router eigrp default Step 4 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with subsequent address family configuration mode commands. Example: apic1(config-eigrp)# vrf member tenant exampleCorp vrf v100 Step 5 autonomous-system asn Enters Autonomous System configuration for EIGRP. Example: apic1(config-eigrp-vrf)# autonomous-system 300 Step 6 address-family {ipv4 | ipv6} unicast Configures an EIGRP policy address family. Example: apic1(config-eigrp-vrf)# address-family ipv4 unicast Step 7 distance internal external Example: Sets EIGRP administrative distance preference for internal and external routes. The distances can be 1 to 255. apic1(config-address-family)# distance 2 5 Step 8 maximum-paths limit Sets EIGRP Maximum Path Limit for the VRF policy template. The limit can be 1 to 32. Example: apic1(config-address-family)# maximum-paths 8 Step 9 Sets EIGRP metric style to wide metric (64 bits). metric version 64bit Example: apic1(config-address-family)# metric version 64bit Step 10 timers active-time minutes Sets EIGRP active timer interval. The range is 1 to 65535 minutes. Example: apic1(config-address-family)# timers active-time 1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 167 Configuring Layer 3 External Connectivity Configuring an EIGRP Interface Step 11 Command or Action Purpose inherit eigrp vrf-policy vrf-policy-name Applies an EIGRP VRF policy to this address family. Example: apic1(config-address-family)# inherit eigrp vrf-policy vrfTemplate3 Examples This example shows how to configure an EIGRP address family and inherit an EIGRP VRF policy. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router eigrp default apic1(config-eigrp)# vrf member tenant exampleCorp vrf v100 apic1(config-eigrp-vrf)# autonomous-system 300 apic1(config-eigrp-vrf)# address-family ipv4 unicast This configuration will affect all leaves where VRF v100 has been deployed apic1(config-address-family)# distance 2 5 This configuration will affect all leaves where VRF v100 has been deployed apic1(config-address-family)# maximum-paths 8 This configuration will affect all leaves where VRF v100 has been deployed apic1(config-address-family)# metric version 64bit This configuration will affect all leaves where VRF v100 has been deployed apic1(config-address-family)# timers active-time 1 This configuration will affect all leaves where VRF v100 has been deployed apic1(config-address-family)# inherit eigrp vrf-policy vrfTemplate3 This template will be inherited on all leaves where VRF v100 has been deployed apic1(config-address-family)# exit apic1(config-eigrp-vrf)# exit apic1(config-eigrp)# exit Configuring an EIGRP Interface Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 interface ethernet slot/port Example: apic1(config-leaf)# interface ethernet 1/21 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 168 Specifies the interface to be configured. Configuring Layer 3 External Connectivity Configuring an EIGRP Interface Step 4 Command or Action Purpose [no] switchport slot/port By default, a port is in Layer 2 trunk mode. If the port is in Layer 3 mode, it must be converted to Layer 2 trunk mode using this command. Example: apic1(config-leaf-if)# no switchport Step 5 [no] vlan-domain member vlan-id Creates and enters the configuration mode for the VLAN domain. Example: apic1(config-leaf-if)# vlan-domain member dom1 Step 6 [no] vrf member tenant exampleCorp vrf vrf-name Associates the interface with a VRF. Example: apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v100 Step 7 [no] {ip | ipv6} address ip-address/mask-length Sets an IP address for the interface. Example: apic1(config-leaf-if)# ip address 181.12.12.1/24 Step 8 [no] {ip | ipv6} router eigrp default Sets router EIGRP policies to default. Example: apic1(config-leaf-if)# ip router eigrp default Step 9 [no] {ip | ipv6} distribute-list eigrp default route-map map-name out Example: apic1(config-leaf-if)# ip distribute-list eigrp default route-map rMapT5 out Step 10 [no] {ip | ipv6} hello-interval eigrp default seconds EIGRP advertises routes that are matched in the route-map specified in the distribute-list command. The route prefixes mentioned in the prefix-list in the route-map can be learned from other protocol sources like BGP, OSPF, Static, Connected. Redistribute route-maps are automatically created based on the distribute-list command. Note that prefixes learned from an EIGRP session running on an another interface on the same switch will not be filtered by the distribute-list and will always be advertised out. Sets EIGRP hello interval time. The range is 1 to 65535 seconds. Example: apic1(config-leaf-if)# ip hello-interval eigrp default 10 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 169 Configuring Layer 3 External Connectivity Configuring an EIGRP Interface Step 11 Command or Action Purpose [no] {ip | ipv6} hold-interval eigrp default seconds Sets EIGRP hold interval time. The range is 1 to 65535 seconds. Example: apic1(config-leaf-if)# ip hold-interval eigrp default 10 Step 12 [no] {ip | ipv6} next-hop-self eigrp default Sets EIGRP next-hop-self flag. Example: apic1(config-leaf-if)# ip next-hop-self eigrp default Step 13 [no] {ip | ipv6} passive-interface eigrp default Set EIGRP passive-interface flag. Example: apic1(config-leaf-if)# ip passive-interface eigrp default Step 14 [no] {ip | ipv6} split-horizon eigrp default Sets EIGRP split-horizon flag. Example: apic1(config-leaf-if)# ip split-horizon eigrp default Step 15 [no] inherit eigrp ip interface-policy if-policy-name Applies an EIGRP interface policy to this interface. Example: apic1(config-leaf-if)# inherit eigrp ip interface-policy ifTemplate5 Step 16 [no] ip summary-address eigrp default ip-prefix Configures route summarization for EIGRP. A summary address can be configured to advertise an aggregated prefix on an EIGRP session. Example: Note apic1(config-leaf-if)# ip summary-address eigrp default 172.10.1.0/24 apic1(config-leaf-if)# ip summary-address eigrp default 2001::/64 A summary address enabled on one interface will also be applied on other EIGRP enabled interfaces on the same VRF on the switch. Examples This example shows how to configure an EIGRP interface. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/21 apic1(config-leaf-if)# no switchport apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v100 apic1(config-leaf-if)# ip address 181.12.12.1/24 apic1(config-leaf-if)# ip router eigrp default apic1(config-leaf-if)# ip distribute-list eigrp default route-map rMapT5 out Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 170 Configuring Layer 3 External Connectivity Configuring Route-Maps distribute list will be updated on all EIGRP interfaces on node 1021 VRF exampleCorp/v100 apic1(config-leaf-if)# ip hello-interval eigrp default 5 apic1(config-leaf-if)# ip hold-interval eigrp default 10 apic1(config-leaf-if)# ip next-hop-self eigrp default apic1(config-leaf-if)# ip passive-interface eigrp default apic1(config-leaf-if)# ip split-horizon eigrp default apic1(config-leaf-if)# inherit eigrp ip interface-policy ifTemplate5 apic1(config-leaf-if)# ip summary-address eigrp default 172.10.1.0/24 apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# exit Configuring Route-Maps Configuring Templates About Route Profiles A route profile specifies the route-control set actions used in import, export, and redistribute route-maps. Route profile templates can be defined either under the tenant or under the tenant VRF. Configuring a Tenant-Scoped Route Profile This procedure creates a tenant-scoped route profile that is used to configure BGP dampening and route redistribution. Before You Begin • Configure a tenant and VRF. • Enable VRF on a leaf. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 [no] template route-profile profile-name tenant Creates a tenant-scoped route profile. tenant-name Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 171 Configuring Layer 3 External Connectivity Configuring Templates Command or Action Purpose Example: apic1(config-leaf)# template route-profile rp1 tenant exampleCorp Step 4 [no] set community {regular | extended} value Sets the BGP community attribute. {none | replace | additive} Example: apic1(config-leaf-template-route-profile)# set community extended 20:22 additive Step 5 [no] set dampening half-life reuse suppress max-suppress-time Example: apic1(config-leaf-template-route-profile)# set dampening 15 750 2000 60 Configures route flap dampening behavior. The parameters are: • half-life—Decay half life, which is the time in minutes after which a penalty is decreased. Once the route has been assigned a penalty, the penalty is decreased by half after the half life period. The range is 1 to 60 minutes. • reuse—A route is unsuppressed (reused) if the penalty for a flapping route decreases enough to fall below this value. The range is 1 to 20000. • suppress—A route is suppressed when its penalty exceeds this limit. The range is 1 to 20000. • max-suppress-time—The maximum time in minutes that a stable route can be suppressed. The range is 1 to 255. Step 6 [no] set local-preference value Sets the BGP local preference value. The range is from 0 to 4294967295. Example: apic1(config-leaf-template-route-profile)# set local-preference 64 Step 7 [no] set metric value Sets the metric for the destination routing protocol. Example: apic1(config-leaf-template-route-profile)# set metric 128 Step 8 [no] set metric-type {type-1 | type-2} Example: apic1(config-leaf-template-route-profile)# set metric-type type-2 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 172 The options are as follows: • type-1—OSPF external type 1 metric • type-2—OSPF external type 2 metric Configuring Layer 3 External Connectivity Configuring Templates Step 9 Command or Action Purpose [no] set tag name Sets the tag value for the destination routing protocol. The name parameter is an unsigned integer. Example: apic1(config-leaf-template-route-profile)# set tag 1111 Step 10 [no] set weight weight Sets the tag value for the destination routing protocol. The weight parameter is an unsigned integer. Example: apic1(config-leaf-template-route-profile)# set weight 20 Examples This example shows how to configure a tenant-scoped route profile. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# template route-profile rp1 tenant exampleCorp This template will be available on all leaves where tenant exampleCorp has a VRF deployment apic1(config-leaf-template-route-profile)# set community extended 20:22 additive apic1(config-leaf-template-route-profile)# set dampening 15 750 2000 60 apic1(config-leaf-template-route-profile)# set local-preference 64 apic1(config-leaf-template-route-profile)# set metric 128 apic1(config-leaf-template-route-profile)# set metric-type type-2 apic1(config-leaf-template-route-profile)# set tag 1111 apic1(config-leaf-template-route-profile)# set weight 20 Configuring a VRF-Scoped Route Profile This procedure creates a VRF-scoped route profile including ‘default-export’ and ‘default-import’. This route profile can be attached to a bridge domain (BD) while ‘matching’ a bridge-domain inside a route map through the inherit route-profile command. Note VRF-scoped route profiles name default-export and default-import values, which are automatically applied on the match statements on the respective export/import route-maps used in the same tenant VRF. Before You Begin • Configure a tenant and VRF. • Enable VRF on a leaf. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 173 Configuring Layer 3 External Connectivity Configuring Templates Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 [no] vrf context tenant tenant-name vrf vrf-name Enables VRF on the leaf and enters VRF configuration mode. Example: apic1(config-leaf)# vrf context tenant exampleCorp vrf vrf1 Step 4 [no] template route-profile profile-name Creates a VRF-scoped route profile. Example: apic1(config-leaf-vrf)# template route-profile default-export Step 5 [no] set community {regular | extended} {no-advertise| no-export|value {none | replace | additive} Sets the BGP community attribute. Example: apic1(config-leaf-vrf-template-route-profile)# set community extended 20:22 additive Step 6 [no] set local-preference value Sets the BGP local preference value. The range is from 0 to 4294967295. Example: apic1(config-tenant-vrf-route-profile)# set local-preference 64 Step 7 [no] set metric value Sets the metric for the destination routing protocol. Example: apic1(config-tenant-vrf-route-profile)# set metric 128 Step 8 [no] set metric-type {type-1 | type-2} Example: apic1(config-tenant-vrf-route-profile)# set metric-type type-2 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 174 The options are as follows: • type-1—OSPF external type 1 metric • type-2—OSPF external type 2 metric Configuring Layer 3 External Connectivity Creating a Route-Map Step 9 Command or Action Purpose [no] set tag name Sets the tag value for the destination routing protocol. The name parameter is an unsigned integer. Example: apic1(config-tenant-vrf-route-profile)# set tag 1111 Step 10 [no] set weight weight Sets the tag value for the destination routing protocol. The weight parameter is an unsigned integer. Example: apic1(config-tenant-vrf-route-profile)# set weight 20 Examples This example shows how to configure a VRF-scoped route profile. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf vrf1 apic1(config-leaf-vrf)# template route-profile default-export apic1(config-leaf-vrf-template-route-profile)# set community extended 20:22 additive apic1(config-leaf-vrf-template-route-profile)# set local-preference 64 apic1(config-leaf-vrf-template-route-profile)# set metric 128 apic1(config-leaf-vrf-template-route-profile)# set metric-type type-2 apic1(config-leaf-vrf-template-route-profile)# set tag 1111 apic1(config-leaf-vrf-template-route-profile)# set weight 20 Creating a Route-Map Route-maps are created with a prefix-list on a per-tenant basis to indicate the bridge domain public subnets to be advertised to external routers. In addition, a prefix-list must be created to allow all transit routes to be advertised to an external router. The prefix-list for transit routes are configured by an administrator. The default behavior is to deny all transit route advertisement to an external router. Before You Begin Configure a tenant and VRF. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 175 Configuring Layer 3 External Connectivity Creating a Route-Map Step 2 Command or Action Purpose leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 [no] vrf context tenant tenant-name vrf vrf-name Configures a tenant VRF on the node. Example: apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 Step 4 [no] router-id ipv4-address Example: apic1(config-leaf-vrf)# router-id 1.2.3.4 Step 5 [no] route-map name (Optional) Assigns a router ID for routing protocols running on the VRF. If you do not assign a router ID, an ID is generated internally that is unique to each leaf switch. Creates a route-map and enters route-map configuration. Example: apic1(config-leaf-vrf)# route-map bgpMap Step 6 [no] ip prefix-list list-name permit prefix/masklen [le {32 | 128}] Creates a prefix-list under the route-map. Example: apic1(config-leaf-vrf-route-map)# ip prefix-list list1 permit 13.13.13.0/24 Step 7 [no] match prefix-list list-name Example: Matches a prefix-list that has already been created and enters the match mode to configure the route-control profile for the prefix-list. apic1(config-leaf-vrf-route-map)# match prefix-list list1 Step 8 [no] set metric value Sets the metric for the destination routing protocol. Example: apic1(config-leaf-vrf-route-map-match)# set metric 128 Step 9 [no] set metric-type {type-1 | type-2} Example: apic1(config-leaf-vrf-route-map-match)# set metric-type type-2 Step 10 [no] set local-preference value Example: apic1(config-leaf-vrf-route-map-match)# set local-preference 64 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 176 The options are as follows: • type-1—OSPF external type 1 metric • type-2—OSPF external type 2 metric Sets the BGP local preference value. The range is from 0 to 4294967295. Configuring Layer 3 External Connectivity Creating a Route-Map Step 11 Command or Action Purpose [no] set community {regular | extended} value {none | replace | additive} Sets the community attribute for a BGP route update. Specify the community-value in aa:nn format. Specify the action as one of the following: Example: apic1(config-leaf-vrf-route-map-match)# set community extended 20:22 additive • additive—Add to existing community • replace—Replace existing community • none—Do not change community Step 12 [no] set tag name Sets the tag value for the destination routing protocol. The name parameter is an unsigned integer. Example: apic1(config-leaf-vrf-route-map-match)# set tag 1111 Step 13 [no] set weight value Specifies the BGP weight for the routing table. Example: apic1(config-leaf-vrf-route-map-match)# set weight 32 Step 14 [no] contract {provider| consumer } contract-name [imported] Add contract, required to leak routes (matching this prefix list) from the VRF. Example: apic1(config-leaf-vrf-route-map-match)# contract provider prov 1 Step 15 [no] match route group group-name [order Matches a route group that has already been created and enters the match mode to configure number ] the route-map. Example: apic1(config-leaf-vrf-route-map)# match route group g1 order 1 Step 16 [no] match bridge-domain bd-name Repeat the steps 8-13 or only step 18 to configure the route map for the route group. See step 17 to inherit the route map instead of inline set actions. Matches a bridge domain in order to export its public subnets through the protocol. Example: apic1(config-leaf-vrf-route-map)# bridge-domain bd1 Step 17 [no] inherit route-profile profile-name Configures route map for bridge domain. Note Example: apic1(config-leaf-vrf-route-map-match)# inherit route-profile rp1 Step 18 [no] bridge-domain-match The route map was already created using the command template route-profile. Configures route map for bridge domain. Example: apic1(config-leaf-vrf-route-map)# no bridge-domain-match Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 177 Configuring Layer 3 External Connectivity Creating a Route-Map Command or Action Purpose Note Disables the bridge domain (BD) match in a route map, eliminating the need to delete the BD configuration from the route map. This is required if there are BDs matched in a route map, and the route map is used to filter out the BD subnets using route group/explicit prefix list. Examples This example shows how to create a route-map and add/match a prefix-list, a community-list, and a bridge-domain. # CREATE A ROUTE-MAP apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 apic1(config-leaf-vrf)# route-map bgpMap # CREATE A PREFIX-LIST apic1(config-leaf-vrf-route-map)# ip prefix-list list1 permit 13.13.13.0/24 apic1(config-leaf-vrf-route-map)# ip prefix-list list1 permit 14.14.14.0/24 # MATCH THE PREFIX-LIST apic1(config-leaf-vrf-route-map)# match prefix-list list1 # CONFIGURE A ROUTE-PROFILE FOR THE apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# PREFIX-LIST set metric 128 set metric-type type-2 set local-preference 64 set community extended 20:22 additive set tag 1111 set weight 32 contract provider prov 1 # CREATE COMMUNITY LIST apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# template community-list standard CL_1 65536:20 tenant exampleCorp # CREATE ROUTE GROUP apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# template route group g1 tenant exampleCorp apic1(config-route-group)# ip prefix permit 15.15.15.0/24 apic1(config-route-group)# community-list standard 65535:20 # MATCH ROUTE GROUP apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 apic1(config-leaf-vrf)# route-map bgpMap apic1(config-leaf-vrf-route-map)# match route group g1 order 1 # CONFIGURE ROUTE PROFILE FOR COMMUNITY-LIST apic1(config-leaf-vrf-route-map-match)# set metric 128 apic1(config-leaf-vrf-route-map-match)# set metric-type type-2 apic1(config-leaf-vrf-route-map-match)# set local-preference 64 apic1(config-leaf-vrf-route-map-match)# set community extended 20:22 additive Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 178 Configuring Layer 3 External Connectivity Configuring Route-Maps in Routing Protocols apic1(config-leaf-vrf-route-map-match)# set tag 1111 apic1(config-leaf-vrf-route-map-match)# set weight 32 # CONFIGURE ROUTE PROFILE ROUTE GROUP apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# apic1(config-leaf-vrf-route-map-match)# set set set set set set metric 128 metric-type type-2 local-preference 64 community extended 20:22 additive tag 1111 weight 32 # Or CREATE A ROUTE PROFILE TEMPLATE AND INHERIT IT FOR ROUTE GROUP apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 apic1(config-leaf-vrf)# template route-profile rp1 apic1(config-leaf-vrf-template-route-profile)# set metric 128 apic1(config-leaf-vrf-template-route-profile)# set metric-type type-2 apic1(config-leaf-vrf-template-route-profile)# set local-preference 64 apic1(config-leaf-vrf-template-route-profile)# set community extended 20:22 additive apic1(config-leaf-vrf-template-route-profile)# set tag 1111 apic1(config-leaf-vrf-template-route-profile)# set weight 32 apic1(config-leaf-vrf-template-route-profile)# exit apic1(config-leaf-vrf)# route-map bgpMap apic1(config-leaf-vrf-route-map)# match route group g1 order 1 apic1(config-leaf-vrf-route-map-match)# inherit route-profile rp1 # CREATE A BRIDGE-DOMAIN apic1# configure apic1(config)# tenant exampleCorp apic1(config-tenant)# vrf context v1 apic1(config-tenant-vrf)# exit apic1(config-tenant)# bridge-domain bd1 apic1(config-tenant-bd)# vrf member v1 apic1(config-tenant-bd)# exit apic1(config-tenant)# interface bridge-domain bd1 apic1(config-tenant-interface)# ip address 13.13.13.1/24 scope public apic1(config-tenant-interface)# exit apic1(config-tenant)# exit # CREATE A ROUTE-PROFILE FOR THE BRIDGE-DOMAIN apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 apic1(config-leaf-vrf)# template route-profile default-export apic1(config-leaf-vrf-template-route-profile)# set metric 128 apic1(config-leaf-vrf-template-route-profile)# set metric-type type-2 apic1(config-leaf-vrf-template-route-profile)# set local-preference 64 apic1(config-leaf-vrf-template-route-profile)# set community extended 20:22 additive apic1(config-leaf-vrf-template-route-profile)# set tag 1111 apic1(config-leaf-vrf-template-route-profile)# set weight 20 apic1(config-leaf-vrf-template-route-profile)# exit # MATCH THE BRIDGE-DOMAIN apic1(config-leaf-vrf)# route-map bgpMap apic1(config-leaf-vrf-route-map)# match bridge-domain bd1 # CONFIGURE A ROUTE-PROFILE FOR THE BRIDGE-DOMAIN apic1(config-leaf-vrf-route-map-match)# inherit route-profile default-export Configuring Route-Maps in Routing Protocols The OSPF, BGP, and EIGRP routing protocols use route-maps to filter routes for import and export. For the general steps required to configure these protocols, see the documentation sections for each. To configure route-maps in these protocols, use the following commands and see the examples. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 179 Configuring Layer 3 External Connectivity Configuring an Export Map (Inter-VRF Route Leak) Protocol Route-Map Command BGP [no] route-map map-name {in | out} OSPF [no] area area-id route-map map-name {in |out } EIGRP [no] ip distribute list default route-map map-name out Examples This example shows how to configure a route-map in BGP, OSPF and EIGRP. # BGP apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router bgp 100 apic1(config-bgp)# vrf member tenant apic1(config-leaf-bgp-vrf)# neighbor apic1(config-leaf-bgp-vrf-neighbor)# apic1(config-leaf-bgp-vrf-neighbor)# apic1(config-leaf-bgp-vrf-neighbor)# apic1(config-leaf-bgp-vrf)# exit apic1(config-bgp)# exit apic1(config-leaf)# exit exampleCorp vrf v1 3.3.3.3 route-map map1 out route-map map2 in exit # OSPF apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router ospf default apic1(config-leaf-ospf)# vrf member tenant exampleCorp vrf v1 apic1(config-leaf-ospf-vrf)# area 0.0.0.1 route-map map1 out apic1(config-leaf-ospf-vrf)# area 0.0.0.1 route-map map2 in apic1(config-leaf-ospf-vrf)# exit apic1(config-leaf-ospf)# exit apic1(config-leaf)# exit #EIGRP apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/3 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# no switchport apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 apic1(config-leaf-if)# ip address 13.13.13.13/24 apic1(config-leaf-if)# ip router eigrp default apic1(config-leaf-if)# ip distribute-list eigrp default route-map map1 out apic1(config-leaf-if)# exit apic1(config-leaf)# exit Configuring an Export Map (Inter-VRF Route Leak) Before You Begin • Create a route-map. • Add prefix-list(s) to the route-map containing prefixes matching routes that need to be leaked. • Match the prefix-list(s) and add the contract(s) to enable the route leak. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 180 Configuring Layer 3 External Connectivity Configuring an Export Map (Inter-VRF Route Leak) Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 [no] vrf context tenant tenant-name vrf vrf-name Configures a tenant VRF on the node. Example: apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 Step 4 [no] export map map-name Example: Configures route-map in this VRF to export (leak) routes from this VRF into consumer VRFs. apic1(config-leaf-vrf)# export map shared-route-map1 Examples This example shows how to create and export a route-map. # CREATE A ROUTE-MAP apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 apic1(config-leaf-vrf)# router-id 1.2.3.4 apic1(config-leaf-vrf)# route-map shared-route-map1 apic1(config-leaf-vrf-route-map)# ip prefix-list list1 permit 13.13.13.0/24 apic1(config-leaf-vrf-route-map)# match prefix-list list1 apic1(config-leaf-vrf-route-map-match)# contract provider prov1 apic1(config-leaf-vrf-route-map-match)# exit apic1(config-leaf-vrf-route-map)# exit apic1(config-leaf-vrf)# exit apic1(config-leaf)# exit # EXPORT THE ROUTE-MAP apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 apic1(config-leaf-vrf)# export map shared-route-map1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 181 Configuring Layer 3 External Connectivity Configuring Bi-Directional Route Forwarding (BFD) Configuring Bi-Directional Route Forwarding (BFD) About BFD Bidirectional Forwarding Detection (BFD) is a detection protocol designed to provide fast forwarding-path failure detection times for media types, encapsulations, topologies, and routing protocols. You can use BFD to detect forwarding path failures at a uniform rate, rather than the variable rates for different protocol hello mechanisms. BFD makes network profiling and planning easier and reconvergence time consistent and predictable. Use Bidirectional Forwarding Detection (BFD) to provide sub-second failure detection times in the forwarding path between ACI fabric border leaf switches configured to support peering router connections. Configuring BFD Globally You can configure the BFD session parameters for all BFD sessions on the device. The BFD session parameters are negotiated between the BFD peers in a three-way handshake. To configure BFD globally, perform the following procedures: • Configure the BFD global configuration settings • Configure an access leaf policy group and inherit the previously created BFD global policies • Associate the previously created leaf policy group onto a leaf switch or group of leaf switches Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 [no] template bfd {ip | ipv6} global-policy-name Creates a BFD policy template. Example: apic1(config)# template bfd ip bfd_global Step 3 [no] echo-address ip-address Example: apic1(config-bfd)# echo-address 192.0.20.123 apic1(config-bfd)# echo-address 34::1/64 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 182 Specifies the IP address to use as the source address for BFD echo packets. Configuring Layer 3 External Connectivity Configuring BFD Globally Step 4 Command or Action Purpose [no] slow-timer milliseconds Configures the slow timer used in the echo function. This value determines how fast BFD starts up a new sessions and at what speed the asynchrounous sessions use for BFD control packets when the echo function is enabled. The slow-timer value is used as the new control packet interval, while the echo packets use the configured BFD intervals. The echo packets are used for link failure detection, while the control packets at the slower rate maintain the BFD session. The range is from 1000 to 30000 milliseconds. Example: apic1(config-bfd)# slow-timer 2000 Step 5 [no] min-tx milliseconds Example: Specifies the interval at which this device wants to send BFD hello messages. The range is 50 to 999 milliseconds. apic1(config-bfd)# min-tx 100 Step 6 [no] min-rx milliseconds Example: Specifies the minimum interval at which this device can accept BFD hello messages from another BFD device. The range is 50 to 999 milliseconds. apic1(config-bfd)# min-rx 70 Step 7 [no] multiplier policy-name Example: apic1(config-bfd)# multiplier 3 Step 8 [no] echo-rx-interval policy-name Example: Specifies the number of missing BFD hello messages from another BFD device before this local device detects a fault in the forwarding path. The range is 1 to 50. Specifies the minimum interval between received BFD echo packets that this system is capable of supporting. The range is 50 to 999 milliseconds. apic1(config-bfd)# echo-rx-interval 500 Step 9 Returns to global configuration mode. exit Example: apic1(config-bfd)# exit Step 10 [no] template leaf-policy-group leaf-policy-name Configures an access leaf policy group. Example: apic1(config)# template leaf-policy-group leaf_pg1 Step 11 [no] inherit bfd {ip | ipv6} global-policy-name Inherits the previously created BFD global policies. Example: apic1(config-leaf-policy-group)# inherit bfd ip bfd_global Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 183 Configuring Layer 3 External Connectivity Configuring BFD Globally Step 12 Command or Action Purpose exit Returns to global configuration mode. Example: apic1(config-leaf-policy-group)# exit Step 13 [no] leaf-profile leaf-profile-name Configures a leaf profile. Example: apic1(config)# leaf-profile leaf_profile1 Step 14 [no] leaf-group leaf-group-name Creates or specifies a group of leaf switches. Example: apic1(config-leaf-profile)# leaf-group leaf_group1 Step 15 [no] leaf-policy-group leaf-policy-name Specifies the previously created leaf policy group to be associated to the leaf switches. Example: apic1(config-leaf-group)# leaf-policy-group leaf_pg1 Step 16 [no] leaf leaf-range Adds one or more leaf switches to the leaf switch group. Example: apic1(config-leaf-group)# leaf 101-102 Examples This example shows how to configure BFD globally and apply it to a group of leaf switches. # CONFIGURE BFD GLOBAL POLICIES apic1# configure apic1(config)# template bfd ip bfd_global apic1(config-bfd)# echo-address 192.0.20.123 apic1(config-bfd)# slow-timer 2000 apic1(config-bfd)# min-tx 100 apic1(config-bfd)# min-rx 70 apic1(config-bfd)# multiplier 3 apic1(config-bfd)# echo-rx-interval 500 apic1(config-bfd)# exit # CONFIGURE AN ACCESS LEAF POLICY GROUP AND INHERIT BFD GLOBAL POLICIES apic1(config)# template leaf-policy-group leaf_pg1 apic1(config-leaf-policy-group)# inherit bfd ip bfd_global apic1(config-leaf-policy-group)# exit # CONFIGURE A LEAF GROUP AND ASSOCIATE THE LEAF POLICY GROUP apic1(config)# leaf-profile leaf_profile1 apic1(config-leaf-profile)# leaf-group leaf_group1 apic1(config-leaf-group)# leaf-policy-group leaf_pg1 apic1(config-leaf-group)# leaf 101-102 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 184 Configuring Layer 3 External Connectivity Overriding Global BFD Settings Overriding Global BFD Settings Configuring BFD Interface Override Policy There are three supported interfaces (routed L3 interfaces, the external SVI interface, and the routed sub-interfaces) on which you can configure an explicit BFD configuration. If you don't want to use the global configuration, yet you want to have an explicit configuration on a given interface, you can create your own global configuration, which gets applied to all the interfaces on a specific switch or set of switches. This interface override configuration should be used if you want more granularity on a specific switch on a specific interface. Before You Begin A tenant has already been created. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Specifies the tenant to be configured. Example: apic1(config)# tenant exampleCorp Step 3 vrf context vrf-name Associates a VRF with the tenant. Example: apic1(config-tenant)# vrf context vrf1 Step 4 Returns to tenant configuration mode. exit Example: apic1(config-tenant-vrf)# exit Step 5 Returns to global configuration mode. exit Example: apic1(config-tenant)# exit Step 6 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 7 [no] vrf context tenant tenant-name vrf vrf-name Configures a tenant VRF on the node. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 185 Configuring Layer 3 External Connectivity Overriding Global BFD Settings Command or Action Purpose Example: apic1(config-leaf)# vrf context tenant exampleCorp vrf vrf1 Step 8 exit Returns to leaf configuration mode. Example: apic1(config-leaf-vrf)# exit Step 9 [no] interface type Enters interface configuration mode. Example: apic1(config-leaf)# interface eth 1/18 Step 10 [no] vrf member tenant tenant-name vrf vrf-name Example: apic1(config-leaf-if)# vrf member tenant exampleCorp vrf vrf1 Step 11 exit Returns to leaf configuration mode. Example: apic1(config-leaf-if)# exit Step 12 [no] template bfd template-name tenant tenant-name Configures a BFD interface policy. Example: apic1(config-leaf)# template bfd bfdIfPol1 tenant exampleCorp Step 13 [no] echo-mode enable Enables or disables the sending of BFD echo packets in addition to BFD control packets. Example: apic1(config-template-bfd-pol)# echo-mode enable Step 14 [no] echo-rx-interval policy-name Example: Specifies the minimum interval between received BFD echo packets that this system is capable of supporting. The range is 50 to 999 milliseconds. apic1(config-template-bfd-pol)# echo-rx-interval 500 Step 15 [no] min-tx milliseconds Example: apic1(config-template-bfd-pol)# min-tx 100 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 186 Specifies the interval at which this device sends BFD hello messages. The range is 50 to 999 milliseconds. Configuring Layer 3 External Connectivity Overriding Global BFD Settings Step 16 Command or Action Purpose [no] min-rx milliseconds Specifies the minimum interval at which this device can accept BFD hello messages from another BFD device. The range is 50 to 999 milliseconds. Example: apic1(config-template-bfd-pol)# min-rx 70 Step 17 [no] multiplier policy-name Example: apic1(config-template-bfd-pol)# multiplier 5 Step 18 [no] optimize subinterface Example: apic1(config-template-bfd-pol)# optimize subinterface Specifies the number of missing BFD hello messages from another BFD device before this local device detects a fault in the forwarding path. The range is 1 to 50. Enables or disables sub-interface optimization. BFD creates sessions for all configured subinterfaces. BFD sets the subinterface with the lowest configured VLAN ID as the master subinterface and that subinterface uses the BFD session parameters of the parent interface. The remaining subinterfaces use the slow timer. If the optimized subinterface session detects an error, BFD marks all subinterfaces on that physical interface as down. Examples This example shows how to configure a BFD override policy and apply it to an interface. apic1# configure apic1(config)# tenant exampleCorp apic1(config-tenant)# vrf context vrf1 apic1(config-tenant-vrf)# exit apic1(config-tenant)# exit apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf vrf1 apic1(config-leaf-vrf)# exit apic1(config-leaf)# interface eth 1/18 apic1(config-leaf-if)# vrf member tenant exampleCorp vrf vrf1 apic1(config-leaf-if)# exit # CONFIGURE BFD INTERFACE OVERRIDE POLICY apic1(config-leaf)# template bfd bfdIfPol1 tenant exampleCorp apic1(config-template-bfd-pol)# echo-mode enable apic1(config-template-bfd-pol)# echo-rx-interval 500 apic1(config-template-bfd-pol)# min-tx 100 apic1(config-template-bfd-pol)# min-rx 70 apic1(config-template-bfd-pol)# multiplier 5 apic1(config-template-bfd-pol)# optimize subinterface Applying the BFD Interface Override Policy to Interfaces You can apply a BFD interface override policy to routed L3 interfaces, the external SVI interface, and the routed sub-interfaces. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 187 Configuring Layer 3 External Connectivity Overriding Global BFD Settings Before You Begin A BFD interface override policy has already been created. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 [no] interface type Example: apic1(config-leaf)# interface Ethernet 1/15 Step 4 Enters interface configuration mode. Supported interfaces are routed L3 interfaces, the external SVI interface, and the routed sub-interfaces. [no] ipv6 address ipv6-address [preferred] Specifies an IP address to be the default source address for traffic from the interface. Example: Note apic1(config-leaf-if)# ipv6 address 2001::10:1/64 preferred Step 5 [no] vrf member tenant tenant-name vrf vrf-name This command is used only if the interface is an IPv6 interface. Attaches the interface to the tenant VRF. Note This command is used only if the interface is a VLAN interface. Example: apic1(config-leaf-if)# vrf member tenant exampleCorp vrf vrf1 Step 6 bfd {ip | ipv6} tenant mode Enables BFD tenant mode. Example: apic1(config-leaf-if)# bfd ip tenant mode Step 7 bfd {ip | ipv6} inherit interface-policy policy-name Inherits the specified BFD interface template policy. Example: apic1(config-leaf-if)# bfd ip inherit interface-policy bfdIfPol1 Step 8 bfd {ip | ipv6} authentication keyed-sha1 keyid Configures BFD authentication as keyed SHA-1. keyid key key Example: apic1(config-leaf-if)# bfd ip authentication keyed-sha1 key 10 key password Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 188 Configuring Layer 3 External Connectivity Enabling BFD on Consumer Protocols Examples This example shows how to inherit the previously created BFD interface policy onto a L3 interface with an IPv4 address. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface apic1(config-leaf-if)# bfd ip apic1(config-leaf-if)# bfd ip apic1(config-leaf-if)# bfd ip eth 1/15 tenant mode inherit interface-policy bfdIfPol1 authentication keyed-sha1 key 10 key password This example shows how to inherit the previously created BFD interface policy onto a L3 interface with an IPv6 address. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/15 apic1(config-leaf-if)# ipv6 address 2001::10:1/64 preferred apic1(config-leaf-if)# bfd ip tenant mode apic1(config-leaf-if)# bfd ip inherit interface-policy bfdIfPol1 apic1(config-leaf-if)# bfd ip authentication keyed-sha1 key 10 key password This example shows how to configure BFD on a VLAN interface with an IPv4 address. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface vlan 15 apic1(config-leaf-if)# vrf member tenant exampleCorp vrf vrf1 apic1(config-leaf-if)# bfd ip tenant mode apic1(config-leaf-if)# bfd ip inherit interface-policy bfdIfPol1 apic1(config-leaf-if)# bfd ip authentication keyed-sha1 key 10 key password This example shows how to configure BFD on a VLAN interface with an IPv6 address. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface vlan 15 apic1(config-leaf-if)# ipv6 address 2001::10:1/64 preferred apic1(config-leaf-if)# vrf member tenant exampleCorp vrf vrf1 apic1(config-leaf-if)# bfd ip tenant mode apic1(config-leaf-if)# bfd ip inherit interface-policy bfdIfPol1 apic1(config-leaf-if)# bfd ip authentication keyed-sha1 key 10 key password Enabling BFD on Consumer Protocols These procedures provide the steps to enable BFD in the four consumer protocols (BGP, EIGRP, OSPF, and Static Routes), which are consumers of the BFD feature. Enabling BFD on the BGP Consumer Protocol Before You Begin A tenant has already been created. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 189 Configuring Layer 3 External Connectivity Enabling BFD on Consumer Protocols Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 bgp-fabric Enters BGP configuration mode for the fabric. Example: apic1(config-bgp-fabric)# bgp-fabric Step 3 asn asn-number Specifies the BGP autonomous system number (ASN). Example: apic1(config-bgp-fabric)# asn 200 Step 4 exit Returns to global configuration mode. Example: apic1(config-bgp-fabric)# exit Step 5 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 6 router bgp asn-number Enters BGP policy configuration. Example: apic1(config-leaf)# router bgp 200 Step 7 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with subsequent policy configuration mode commands. Example: apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 Step 8 neighbor ip-address[ /masklength ] Specifies the IP address of the neighbor. The mask length must be 32. Example: apic1(config-leaf-bgp-vrf)# neighbor 1.2.3.4 Step 9 [no] bfd enable Example: apic1(config-leaf-bgp-vrf-neighbor)# bfd enable Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 190 Enables or disables BFD on the BGP consumer protocol. Configuring Layer 3 External Connectivity Enabling BFD on Consumer Protocols Examples This example shows how to enable BFD on the BGP consumer protocol. apic1# configure apic1(config)# bgp-fabric apic1(config-bgp-fabric)# asn 200 apic1(config-bgp-fabric)# exit apic1(config)# leaf 101 apic1(config-leaf)# router bgp 200 apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 apic1(config-leaf-bgp-vrf)# neighbor 1.2.3.4 apic1(config-leaf-bgp-vrf-neighbor)# bfd enable Enabling BFD on the EIGRP Consumer Protocol Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 [no] interface type Enters interface configuration mode. Example: apic1(config-leaf)# interface Ethernet 1/15 Step 4 [no] {ip | ipv6} bfd eigrp enable Enables or disables BFD on the EIGRP consumer protocol. Example: apic1(config-leaf-if)# ip bfd eigrp enable Examples This example shows how to enable BFD on the EIGRP consumer protocol. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/15 apic1(config-leaf-if)# ip bfd eigrp enable Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 191 Configuring Layer 3 External Connectivity Enabling BFD on Consumer Protocols Enabling BFD on the OSPF Consumer Protocol Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 [no] interface type Enters interface configuration mode. Example: apic1(config-leaf)# interface vlan 123 Step 4 [no] ip ospf bfd enable Enables or disables BFD on the OSPF consumer protocol. Example: apic1(config-leaf-if)# ip ospf bfd enable Examples This example shows how to enable BFD on the OSPF consumer protocol. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface vlan 123 apic1(config-leaf-if)# ip ospf bfd enable Enabling BFD on the Static Route Consumer Protocol Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Example: apic1(config)# leaf 101 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 192 Specifies the leaf to be configured. Configuring Layer 3 External Connectivity Configuring Layer 3 Multicast Step 3 Command or Action Purpose [no] vrf context tenant tenant-name vrf vrf-name Configures a tenant VRF on the node. Example: apic1(config-leaf)# vrf context tenant exampleCorp vrf vrf1 Step 4 [no] {ip | ipv6} route ip-prefix/masklen next-hop-address bfd Enables or disables BFD on the static route consumer protocol. Example: apic1(config-leaf-vrf)# ip route 10.0.0.1/16 10.0.0.5 bfd Examples This example shows how to enable BFD on the static route consumer protocol. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf vrf1 apic1(config-leaf-vrf)# ip route 10.0.0.1/16 10.0.0.5 bfd Configuring Layer 3 Multicast Layer 3 Multicast In the ACI fabric, most unicast and multicast routing operate together on the same border leaf switches, with the multicast protocol operating over the unicast routing protocols. In this architecture, only the border leaf switches run the full Protocol Independent Multicast (PIM) protocol. Non-border leaf switches run PIM in a passive mode on the interfaces. They do not peer with any other PIM routers. The border leaf switches peer with other PIM routers connected to them over L3 Outs and also with each other. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 193 Configuring Layer 3 External Connectivity Guidelines for Configuring Layer 3 Multicast The following figure shows the border leaf (BL) switches (BL1 and BL2) connecting to routers (R1 and R2) in the multicast cloud. Each virtual routing and forwarding (VRF) in the fabric that requires multicast routing will peer separately with external multicast routers. Figure 18: Overview of Multicast Cloud Guidelines for Configuring Layer 3 Multicast See the following guidelines: • The Layer 3 multicast configuration is done at the VRF level so protocols function within the VRF and multicast is enabled in a VRF, and each multicast VRF can be turned on or off independently. • Once a VRF is enabled for multicast, the individual bridge domains (BDs) and L3 Outs under the enabled VRF can be enabled for multicast configuration. By default, multicast is disabled in all BDs and Layer 3 Outs. • Layer 3 multicast is not currently supported on VRFs that are configured with a shared L3 Out. • Any Source Multicast (ASM) and Source-Specific Multicast (SSM) are supported. • Bidirectional PIM, Rendezvous Point (RP) within the ACI fabric, and PIM IPv6 are currently not supported. • IGMP snooping cannot be disabled on pervasive bridge domains with multicast routing enabled. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 194 Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast • Multicast routers are not supported in pervasive bridge domains. • The Layer 3 multicast feature is supported on the following -EX model leaf switches: • N9K-93180YC-EX • N9K-93108TC-EX • N9K-93180LC-EX • Layer 3 Out ports and sub-interfaces are supported while external SVIs are not supported. Since external SVIs are not supported, PIM cannot be enabled in L3-VPC. • For Layer 3 multicast support for multipod, when the ingress leaf switch receives a packet from a source attached on a bridge domain that is enabled for multicast routing, the ingress leaf switch sends only a routed VRF copy to the fabric (routed implies that the TTL is decremented by 1, and the source-mac is rewritten with a pervasive subnet MAC). The egress leaf switch also routes the packet into receivers in all the relevant bridge domains. Therefore, if a receiver is on the same bridge domain as the source, but on a different leaf switch than the source, that receiver continues to get a routed copy, even though it is in the same bridge domain. For more information, see details about layer 3 multicast support for multipod that leverages existing Layer 2 design, at the following link Adding Pods. • Layer 3 multicast is not supported with FEX. Multicast sources or receivers connected to FEX ports are not supported. Note When you configure Layer 3 Outside (L3Out) connections to external routers, or multipod connections through an Inter-Pod Network (IPN), it is critical that the MTU be set appropriately on both sides. On some platforms, such as ACI, Cisco NX-OS, and Cisco IOS, the configurable MTU value takes into account packet headers (resulting in a max packet size to be set as 9000 bytes), whereas other platforms such as IOS-XR configure the MTU value exclusive of packet headers (resulting in a max packet size of 8986 bytes). For the appropriate MTU values for each platform, see the relevant configuration guides. Cisco highly recommends you test the MTU using CLI-based commands. For example, on the Cisco NX-OS CLI, use a command such as ping 1.1.1.1 df-bit packet-size 9000 source-interface ethernet 1/1. Configuration Steps for Layer 3 Multicast The following sections show the configuration steps for layer 3 Multicast. The steps are as follows: 1 Configure PIM options on the tenant VRF. 2 Configure IGMP options for the VRF. 3 Configure an L3 Out for the tenant, enable PIM, and configure the leaf interface. 4 Enable PIM in the desired bridge domains. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 195 Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast Configuring PIM Options for Layer 3 Multicast Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Specifies the tenant to be configured. Example: apic1(config)# tenant exampleCorp Step 3 vrf context vrf-name Associates a VRF with the tenant. Example: apic1(config-tenant)# vrf context exampleCorp_vrf1 Step 4 [no] ip pim Configures Protocol Independent Multicast (PIM). Example: apic1(config-tenant-vrf)# ip pim Step 5 [no] ip pim auto-rp {forward [listen] | listen | mapping-agent-policy mapping-agent-policy-name} Example: apic1(config-tenant-vrf)# ip pim auto-rp forward listen Step 6 [no] ip pim bsr {forward [listen] | listen (Optional) | bsr-policy mapping-agent-policy-name} Configures PIM bootstrap router (BSR) options. BSR performs similarly to auto-RP in that it uses candidate routers for the RP function and for relaying the RP Example: apic1(config-tenant-vrf)# ip pim bsr information for a group. RP information is distributed forward listen through BSR messages, which are carried within PIM messages. You can choose to forward Bootstrap/Candidate-RP messages, listen to Bootstrap/Candidate-RP messages, or associate a route-map policy for filtering BSR messages. Step 7 [no] ip pim fast-convergence Example: apic1(config-tenant-vrf)# ip pim fast-convergence Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 196 (Optional) Configures PIM auto-RP (Rendezvous Point) options. Auto-RP automates the distribution of group-to-RP mappings in a PIM network. You can choose to forward auto-RP messages, listen to auto-RP messages, or associate a route-map policy for filtering mapping agent messages. (Optional) Enables the PIM fast convergence feature, which allows the switch to discover unresponsive neighbors more quickly. Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast Step 8 Command or Action Purpose [no] ip pim mtu mtu-size (Optional) Configures the maximum size of a PIM message. The range is 1500 to 65536 bytes. Example: apic1(config-tenant-vrf)# ip pim mtu 1500 Step 9 [no] ip pim register-policy register-policy-name (Optional) Specifies the name of a policy for filtering register messages. Example: apic1(config-tenant-vrf)# ip pim register-policy regPolicy1 Step 10 [no] ip pim register-rate-limit mtu-size Example: (Optional) Specifies a rate limit for PIM data registers. The range is 0 to 65535 packets per second. apic1(config-tenant-vrf)# ip pim register-rate-limit 1024 Step 11 [no] ip pim register-source ip-address (Optional) Configures a source IP address for PIM messages. Example: apic1(config-tenant-vrf)# ip pim register-source 192.0.20.123 Step 12 [no] ip pim rp-address ip-address [route-map route-map-name] (Optional) Configures a static route processor (RP) address for a multicast group range. Example: apic1(config-tenant-vrf)# ip pim rp-address 192.0.20.99 Step 13 [no] ip pim sg-expiry-timer ip-address [sg-list route-map-name] Example: apic1(config-tenant-vrf)# ip pim sg-expiry-timer 4096 Step 14 [no] ip pim ssm route-map route-map-name Example: apic1(config-tenant-vrf)# ip pim ssm route-map SSMRtMap Step 15 [no] ip pim state-limit max-entries [reserved route-map-name [maximum-reserve-state-entries]] Example: apic1(config-tenant-vrf)# ip pim state-limit 100000 reserved myReservedPolicy 40000 (Optional) Configures the (S, G) expiry timer interval for PIM sparse mode (PIM-SM) (S, G) multicast routes. The range is 180 to 604801 seconds. The optional sg-list parameter specifies S,G values to which the timer applies. The default is 4096. (Optional) Configures Source Specific Multicast (SSM), which is an extension of IP multicast in which datagram traffic is forwarded to receivers from only those multicast sources that the receivers have explicitly joined. The route-map policy lists the group prefixes. (Optional) Configures a maximum number of PIM state entries in the current VRF instance. The range is 0 to 4294967295 maximum state entries. You can optionally specify a number of state entries to be reserved for the routes specified in a policy map and you can specify the maximum reserved (*, G) and (S, G) entries allowed in this VRF. This number must be Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 197 Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast Command or Action Purpose less than or equal to the maximum states allowed. The range is from 1 to 4294967295. Step 16 [no] ip pim use-shared-tree-only group-list policy-name Example: (Optional) Creates the PIM (*, G) state only (where no source state is created). The policy defines the group prefixes where this feature is applied. apic1(config-tenant-vrf)# ip pim use-shared-tree-only group-list myGroup1 Step 17 exit Returns to tenant configuration mode. Example: apic1(config-tenant-vrf)# exit What to Do Next Configure IGMP options for the VRF. Configuring IGMP Options on the VRF for Layer 3 Multicast Before You Begin Configure PIM options on the tenant VRF. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Specifies the tenant to be configured. Example: apic1(config)# tenant exampleCorp Step 3 vrf context vrf-name Associates a VRF with the tenant. Example: apic1(config-tenant)# vrf context vrf1 Step 4 [no] ip igmp Example: apic1(config-tenant-vrf)# ip igmp Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 198 Enables Internet Group Management Protocol (IGMP). Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast Step 5 Command or Action Purpose exit Returns to tenant configuration mode. Example: apic1(config-tenant-vrf)# exit Step 6 interface bridge-domain bd-name Enters tenant interface configuration mode to configure the bridge domain. Example: apic1(config-tenant)# interface bridge-domain exampleCorp_bd1 Step 7 Enables IP multicast routing on the interface. [no] ip multicast Example: apic1(config-tenant-interface)# ip multicast Step 8 [no] ip igmp allow-v3-asm Allows filtering for source addresses in IGMPv3 reports for Any Source Multicast (ASM) groups. Example: apic1(config-tenant-interface)# ip igmp allow-v3-asm Step 9 [no] ip igmp fast-leave Example: apic1(config-tenant-interface)# ip igmp fast-leave Step 10 [no] ip igmp group-timeout seconds Example: Enables IP IGMP snooping fast leave processing. This feature supports IGMPv2 hosts that cannot be explicitly tracked because of the host report suppression mechanism of the IGMPv2 protocol. When you enable fast leave, the IGMP software assumes that no more than one host is present on each port. Sets the group membership timeout for IGMPv2. The range is 3 to 65535 seconds. The default is 260 seconds. apic1(config-tenant-interface)# ip igmp group-timeout 260 Step 11 [no] ip igmp inherit interface-policy policy-name Associates a IGMP interface policy to this interface. Example: apic1(config-tenant-interface)# ip igmp inherit interface-policy MyIfPolicy Step 12 [no] ip igmp join-group route-map route-map-name Statically binds one or more multicast groups to the interface. The route-map policy lists the group prefixes, group ranges, and source prefixes. Example: apic1(config-tenant-interface)# ip igmp join-group route-map MyGroupsRMap Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 199 Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast Command or Action Step 13 Purpose [no] ip igmp last-member-query-count Sets the number of times that the software sends an IGMP query in response to a host leave message. The count range is 1 to 5 queries. The default is 2 queries. Example: apic1(config-tenant-interface)# ip igmp last-member-query-count 2 Step 14 [no] ip igmp last-member-query-response-time seconds Sets the query interval waited after sending membership reports before the software deletes the group state. The range is 1 to 25 seconds. The default is 1 second. Example: apic1(config-tenant-interface)# ip igmp last-member-query-response-time 1 Step 15 [no] ip igmp querier-timeout seconds Example: apic1(config-tenant-interface)# ip igmp querier-timeout 255 Step 16 [no] ip igmp query-interval seconds Example: apic1(config-tenant-interface)# ip igmp query-interval 125 Sets the number of seconds that the software waits after the previous querier has stopped querying and before it takes over as the querier. The range is 1 to 65535 seconds. The default is 255 seconds. Sets the frequency at which the software sends IGMP host query messages. You can tune the number of IGMP messages on the network by setting a larger value so that the software sends IGMP queries less often. The range is 1 to 18000 seconds. The default is 125 seconds. Step 17 [no] ip igmp query-max-response-time Sets the response time advertised in IGMP queries. You can tune the burstiness of IGMP messages on the seconds network by setting a larger value so that host responses are spread out over a longer time. This value must be Example: apic1(config-tenant-interface)# ip less than the query interval. The range is 1 to 25 igmp query-max-response-time 10 seconds. The default is 10 seconds. Step 18 [no] ip igmp report-link-local-groups Example: apic1(config-tenant-interface)# ip igmp report-link-local-groups Step 19 Enables sending reports for groups in 224.0.0.0/24. Link local addresses are used only by protocols on the local network. Reports are always sent for nonlink local groups. By default, reports are not sent for link local groups. [no] ip igmp report-policy policy-name Configures an access policy for IGMP reports that is based on a route-map policy. Example: apic1(config-tenant-interface)# ip igmp report-policy MyReportPolicy Step 20 [no] ip igmp robustness-variable value Sets the robustness variable to compensate for packet loss on a congested network. The robustness value is used by the IGMP software to determine the number Example: apic1(config-tenant-interface)# ip of times to send messages. You can use a larger value igmp robustness-variable 2 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 200 Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast Command or Action Purpose for a lossy network. The range is 1 to 7. The default is 2. Step 21 Enables IGMP snooping for the interface. [no] ip igmp snooping Example: apic1(config-tenant-interface)# ip igmp snooping Step 22 [no] ip igmp snooping fast-leave Example: apic1(config-tenant-interface)# ip igmp snooping fast-leave Step 23 [no] ip igmp snooping last-member-query-interval Example: Enables the software to remove the group state when it receives an IGMP Leave report without sending an IGMP query message. This parameter is used for IGMPv2 hosts when no more than one host is present on each port. Sets a time interval in seconds after which the group is removed from the associated port if no hosts respond to an IGMP query message. The range is 1 to 25 seconds. The default is 5 seconds. apic1(config-tenant-interface)# ip igmp snooping last-member-query-interval 5 Step 24 [no] ip igmp snooping policy policy-name Associates the bridge domain with an IGMP snooping policy. Example: apic1(config-tenant-interface)# ip igmp snooping policy MySnoopingPolicy Step 25 [no] ip igmp snooping querier Example: apic1(config-tenant-interface)# ip igmp snooping querier Step 26 [no] ip igmp snooping query-interval seconds Example: Enables an IP IGMP snooping querier, which sends out periodic IGMP queries that trigger IGMP report messages from hosts who want to receive IP multicast traffic. IGMP snooping listens to these IGMP reports to establish appropriate forwarding. Configures a snooping query interval when you do not enable PIM because multicast traffic does not need to be routed. The range is 1 to 18000 seconds. The default is 125 seconds. apic1(config-tenant-interface)# ip igmp snooping query-interval 125 Step 27 [no] ip igmp snooping query-max-response-time seconds Example: Configures a snooping maximum response time for query messages when you do not enable PIM because multicast traffic does not need to be routed. The range is 1 to 25 seconds. The default is 10 seconds. apic1(config-tenant-interface)# ip igmp snooping query-max-response-time 10 Step 28 [no] ip igmp snooping startup-query-count count Configures snooping for a number of queries sent at startup when you do not enable PIM because multicast traffic does not need to be routed. The range is 1 to 10 queries. The default is 5 queries. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 201 Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast Command or Action Purpose Example: apic1(config-tenant-interface)# ip igmp snooping startup-query-count 5 Step 29 [no] ip igmp snooping startup-query-interval seconds Example: Configures a snooping query interval at startup when you do not enable PIM because multicast traffic does not need to be routed. The range is 1 to 18000 seconds. The default is 15000 seconds. apic1(config-tenant-interface)# ip igmp snooping startup-query-interval 15000 Step 30 [no] ip igmp startup-query-count count Sets the number of queries sent at startup that are separated by the startup query interval. The range is 1 to 10 queries. The default is 2 queries. Example: apic1(config-tenant-interface)# ip igmp startup-query-count 2 Step 31 [no] ip igmp startup-query-interval seconds Example: apic1(config-tenant-interface)# ip igmp startup-query-interval 31 Step 32 [no] ip igmp state-limit max-states [reserved route-map-name [max-reserved-gsg-entries]] Example: apic1(config-tenant-interface)# ip igmp state-limit 100000 reserved myReservedPolicy 40000 Step 33 [no] ip igmp static-oif route-map route-map-name Example: Sets the query interval used when the software starts up. By default, this interval is shorter than the query interval so that the software can establish the group state as quickly as possible. The range is 1 to 18000 seconds. The default is 260 seconds. The default is 31 seconds. Configures a per interface limit on the number of mroutes states created as a result of IGMP membership reports (IGMP joins). The range of states allowed is 1 to 4294967295 states. You can optionally specify a number of state entries to be reserved for the routes specified in a policy map and you can specify the maximum reserved (*, G) and (S, G) entries allowed on the interface. The number of reserved states must be less than or equal to the maximum states allowed. The range is from 1 to 4294967295. Statically binds a multicast group to the outgoing interface (OIF), which is handled by the device hardware. The route map defines the group prefixes where this feature is applied. apic1(config-tenant-interface)# ip igmp static-oif route-map MyOifMap Step 34 [no] ip igmp version {v1 | v2 | v3} Configures the IGMP version number for the interface. The default version is v2. Example: apic1(config-tenant-interface)# ip igmp version v3 Step 35 exit Example: apic1(config-tenant-interface)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 202 Returns to tenant configuration mode. Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast What to Do Next Configure an L3 Out for the tenant, enable PIM, and configure the leaf interface. Configuring an L3 Out for Layer 3 Multicast Before You Begin • Configure PIM options on the tenant VRF. • Configure IGMP on the tenant VRF. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Specifies the tenant to be configured. Example: apic1(config)# tenant exampleCorp Step 3 l3out l3out-name Configures an L3 Out interface on the tenant. Example: apic1(config-tenant)# l3out exampleCorp_l3out Step 4 Enables PIM on the interface. ip pim Example: apic1(config-tenant-l3out)# ip pim Step 5 Returns to tenant configuration mode. exit Example: apic1(config-tenant-l3out)# Step 6 Returns to global configuration mode. exit Example: apic1(config-tenant)# exit Step 7 leaf node-id Enters leaf configuration mode. Example: apic1(config)# leaf 101 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 203 Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast Step 8 Command or Action Purpose interface ethernet slot/port Specifies the interface to be configured. Example: apic1(config-leaf)# interface ethernet 1/3 Step 9 [no] ip igmp allow-v3-asm Allows filtering for source addresses in IGMPv3 reports for Any Source Multicast (ASM) groups. Example: apic1(config-leaf-if)# ip igmp allow-v3-asm Step 10 [no] ip igmp fast-leave Example: apic1(config-leaf-if)# ip igmp fast-leave Step 11 [no] ip igmp group-timeout seconds Example: Enables IP IGMP snooping fast leave processing. This feature supports IGMPv2 hosts that cannot be explicitly tracked because of the host report suppression mechanism of the IGMPv2 protocol. When you enable fast leave, the IGMP software assumes that no more than one host is present on each port. Sets the group membership timeout for IGMPv2. The range is 3 to 65535 seconds. The default is 260 seconds. apic1(config-leaf-if)# ip igmp group-timeout 260 Step 12 [no] ip igmp inherit interface-policy policy-name Associates a IGMP interface policy to this interface. Example: apic1(config-leaf-if)# ip igmp inherit interface-policy MyIfPolicy Step 13 [no] ip igmp join-group route-map route-map-name Statically binds one or more multicast groups to the interface. The route-map policy lists the group prefixes, group ranges, and source prefixes. Example: apic1(config-leaf-if)# ip igmp join-group route-map MyGroupsRMap Step 14 [no] ip igmp last-member-query-count Sets the number of times that the software sends an IGMP query in response to a host leave message. The count range is 1 to 5 queries. The default is 2 queries. Example: apic1(config-leaf-if)# ip igmp last-member-query-count 2 Step 15 [no] ip igmp last-member-query-response-time seconds Example: apic1(config-leaf-if)# ip igmp last-member-query-response-time 1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 204 Sets the query interval waited after sending membership reports before the software deletes the group state. The range is 1 to 25 seconds. The default is 1 second. Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast Step 16 Command or Action Purpose [no] ip igmp querier-timeout seconds Sets the number of seconds that the software waits after the previous querier has stopped querying and before it takes over as the querier. The range is 1 to 65535 seconds. The default is 255 seconds. Example: apic1(config-leaf-if)# ip igmp querier-timeout 255 Step 17 [no] ip igmp query-interval seconds Example: apic1(config-leaf-if)# ip igmp query-interval 125 Sets the frequency at which the software sends IGMP host query messages. You can tune the number of IGMP messages on the network by setting a larger value so that the software sends IGMP queries less often. The range is 1 to 18000 seconds. The default is 125 seconds. Step 18 [no] ip igmp query-max-response-time Sets the response time advertised in IGMP queries. You can tune the burstiness of IGMP messages on the seconds network by setting a larger value so that host responses are spread out over a longer time. This value must be Example: less than the query interval. The range is 1 to 25 apic1(config-leaf-if)# ip igmp query-max-response-time 10 seconds. The default is 10 seconds. Step 19 [no] ip igmp report-link-local-groups Enables sending reports for groups in 224.0.0.0/24. Link local addresses are used only by protocols on the local network. Reports are always sent for nonlink Example: local groups. By default, reports are not sent for link apic1(config-leaf-if)# ip igmp report-link-local-groups local groups. Step 20 [no] ip igmp report-policy policy-name Configures an access policy for IGMP reports that is based on a route-map policy. Example: apic1(config-leaf-if)# ip igmp report-policy MyReportPolicy Step 21 [no] ip igmp robustness-variable value Sets the robustness variable to compensate for packet loss on a congested network. The robustness value is used by the IGMP software to determine the number Example: of times to send messages. You can use a larger value apic1(config-leaf-if)# ip igmp robustness-variable 2 for a lossy network. The range is 1 to 7. The default is 2. Step 22 [no] ip igmp startup-query-count count Sets the number of queries sent at startup that are separated by the startup query interval. The range is 1 to 10 queries. The default is 2 queries. Example: apic1(config-leaf-if)# ip igmp startup-query-count 2 Step 23 [no] ip igmp startup-query-interval seconds Example: apic1(config-leaf-if)# ip igmp startup-query-interval 31 Sets the query interval used when the software starts up. By default, this interval is shorter than the query interval so that the software can establish the group state as quickly as possible. The range is 1 to 18000 seconds. The default is 260 seconds. The default is 31 seconds. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 205 Configuring Layer 3 External Connectivity Configuration Steps for Layer 3 Multicast Step 24 Command or Action Purpose [no] ip igmp state-limit max-states [reserved route-map-name [max-reserved-gsg-entries]] Configures a per interface limit on the number of mroutes states created as a result of IGMP membership reports (IGMP joins). The range of states allowed is 1 to 4294967295 states. You can optionally specify a number of state entries to be reserved for the routes specified in a policy map and you can specify the maximum reserved (*, G) and (S, G) entries allowed on the interface. The number of reserved states must be less than or equal to the maximum states allowed. The range is from 1 to 4294967295. Example: apic1(config-leaf-if)# ip igmp state-limit 100000 reserved myReservedPolicy 40000 Step 25 [no] ip igmp static-oif route-map route-map-name Example: Statically binds a multicast group to the outgoing interface (OIF), which is handled by the device hardware. The route map defines the group prefixes where this feature is applied. apic1(config-leaf-if)# ip igmp static-oif route-map MyOifMap Step 26 [no] ip igmp version {v1 | v2 | v3} Configures the IGMP version number for the interface. The default version is v2. Example: apic1(config-leaf-if)# ip igmp version v3 Step 27 exit Returns to tenant configuration mode. Example: apic1(config-leaf-if)# exit Example: Configuring Layer 3 Multicast # CONFIGURE PIM OPTIONS ON A TENANT VRF apic1# configure apic1(config)# tenant exampleCorp apic1(config-tenant)# vrf context exampleCorp_vrf1 apic1(config-tenant-vrf)# ip pim apic1(config-tenant-vrf)# ip pim fast-convergence apic1(config-tenant-vrf)# ip pim bsr forward # ENABLE AND CONFIGURE IGMP ON THE TENANT VRF AND BRIDGE DOMAIN apic1(config-tenant-vrf)# ip igmp apic1(config-tenant-vrf)# exit apic1(config-tenant)# interface bridge-domain exampleCorp_bd apic1(config-tenant-interface)# ip multicast apic1(config-tenant-interface)# ip igmp allow-v3-asm apic1(config-tenant-interface)# ip igmp fast-leave apic1(config-tenant-interface)# exit # CREATE AN L3OUT AND CONFIGURE PIM apic1(config-tenant)# l3out exampleCorp_l3out apic1(config-tenant-l3out)# ip pim Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 206 Configuring Layer 3 External Connectivity Configuring External-L3 EPGs apic1(config-tenant-l3out)# exit apic1(config-tenant)# exit # CONFIGURE AN EXTERNAL INTERFACE AND CONFIGURE IGMP ON THE INTERFACE apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/125 apic1(config-leaf-if)# ip igmp fast-leave apic1(config-leaf-if)# ip-igmp join-group Configuring External-L3 EPGs External-L3 EPGs are classified under a tenant VRF. In the CLI, an external-l3 EPG is defined in the tenant mode and is deployed to individual nodes. You have the flexibility to place external-l3 EPGs in a select set of nodes instead of all nodes in a VRF. Each external-l3 EPG can be a producer/consumer of multiple contracts, and each external-l3 EPG has its own QoS policy for DSCP marking and queuing priority within the fabric. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Enters the tenant configuration mode. Example: apic1(config)# tenant exampleCorp Step 3 external-l3 epg epg-name Enters the external-l3 EPG configuration mode. Example: apic1(config-tenant)# external-l3 epg epgExtern1 Step 4 vrf member vrf-name Associates the EPG with a VRF. Example: apic1(config-tenant-l3ext-epg)# vrf member v1 Step 5 match {ip | ipv6} ip-address/masklength Creates a rule to match a subnet. Example: apic1(config-tenant-l3ext-epg)# match ip 192.0.20.0/24 apic1(config-tenant-l3ext-epg)# match ipv6 2001::1/64 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 207 Configuring Layer 3 External Connectivity Configuring External-L3 EPGs Step 6 Command or Action Purpose set qos-class class Specifies the QOS level for the EPG. Example: apic1(config-tenant-l3ext-epg)# set qos-class level1 Step 7 set dscp dscp-value Specifies the DSCP value for the EPG. Example: apic1(config-tenant-l3ext-epg)# set dscp af31 Step 8 contract consumer contract-name Specifies the consumer contract for the EPG. Example: apic1(config-tenant-l3ext-epg)# contract consumer cConsumer1 Step 9 contract provider contract-name Specifies the provider contract for the EPG. Example: apic1(config-tenant-l3ext-epg)# contract provider cProvider1 Step 10 contract deny contract-name Specifies a deny contract for the EPG. Example: apic1(config-tenant-l3ext-epg)# contract deny cDeny1 Step 11 exit Example: apic1(config-tenant-l3ext-epg)# exit Step 12 exit Example: apic1(config-tenant)# exit Step 13 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 14 vrf context tenant tenant-name vrf vrf-name Configures a tenant VRF on the node. Example: apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 Step 15 external-l3 epg epg-name Example: apic1(config-leaf-vrf)# external-l3 epg epgExtern1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 208 Associates the external layer 3 EPG on the VRF. Configuring Layer 3 External Connectivity Configuring Layer 3 External Connectivity Using the Named Mode Examples This example shows how to configure an external layer 3 EPG and to deploy the EPG on a leaf. apic1# configure apic1(config)# tenant exampleCorp # CONFIGURE EXTERNAL L3 EPG apic1(config-tenant)# external-l3 epg epgExtern1 apic1(config-tenant-l3ext-epg)# vrf member v1 apic1(config-tenant-l3ext-epg)# match ip 192.0.20.0/24 apic1(config-tenant-l3ext-epg)# match ipv6 2001::1/64 apic1(config-tenant-l3ext-epg)# set qos-class level1 apic1(config-tenant-l3ext-epg)# set dscp af31 apic1(config-tenant-l3ext-epg)# contract consumer cConsumer1 apic1(config-tenant-l3ext-epg)# contract provider cProvider1 apic1(config-tenant-l3ext-epg)# contract deny cDeny1 apic1(config-tenant-l3ext-epg)# exit apic1(config-tenant)# exit # DEPLOY EXTERNAL L3 EPG ON A LEAF apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 apic1(config-leaf-vrf)# external-l3 epg epgExtern1 Configuring Layer 3 External Connectivity Using the Named Mode Creating a Named L3Out Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 tenant tenant-name Enters the tenant configuration mode. Example: apic1(config)# tenant exampleCorp Step 3 vrf context vrf-name Associates the tenant with a VRF. Example: apic1(config-tenant)# vrf context v1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 209 Configuring Layer 3 External Connectivity Creating a Named L3Out Step 4 Command or Action Purpose l3out l3out-name Creates a named L3Out. Example: apic1(config-tenant)# l3out out1 Step 5 vrf member vrf-name Associates the L3Out with the tenant VRF. Example: apic1(config-tenant-l3out)# vrf member v1 Step 6 exit Returns to tenant configuration mode. Example: apic1(config-tenant-l3out)# exit Step 7 exit Returns to global configuration mode. Example: apic1(config-tenant)# exit Step 8 leaf node-id node Example: apic1(config)# leaf 101 Step 9 vrf context tenant tenant-name vrf vrf-name l3out Configures a tenant VRF on the node. l3out-name Example: apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 l3out out1 Step 10 [no] router-id ipv4-address Assigns a router ID for routing protocols running on the VRF. Example: apic1(config-leaf-vrf)# router-id 1.2.3.4 Step 11 [no] {ip | ipv6} route ip-prefix/masklen next-hop-address [preferred] Configures static route information for the VRF. Example: apic1(config-leaf-vrf)# ip route 21.1.1.1/32 32.1.1.1 apic1(config-leaf-vrf)# ipv6 route 5001::1/128 6002::1 Examples This example shows how to create a named L3Out under the tenant, assign it to the tenant VRF, and deploy it on the border leaf switch. apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 210 Configuring Layer 3 External Connectivity Configuring Layer 3 Interfaces for a Named L3Out apic1(config)# tenant exampleCorp apic1(config-tenant)# vrf context v1 apic1(config-tenant)# l3out out1 apic1(config-tenant-l3out)# vrf member v1 apic1(config-tenant-l3out)# exit apic1(config-tenant)# exit apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 l3out out1 apic1(config-leaf-vrf)# router-id 1.2.3.4 apic1(config-leaf-vrf)# ip route 21.1.1.1/32 32.1.1.1 What to Do Next Configure layer 3 interfaces for the named L3Out. Configuring Layer 3 Interfaces for a Named L3Out This procedure shows how to configure a layer 3 port interface to a named L3Out. The examples show how to configure a subinterface or SVI to a named L3Out. • A given interface can be added to multiple L3Outs by providing multiple L3Out names after the l3out keyword. • An SVI can be configured using the switchport trunk allowed vlan command under any of the following interface types: ◦interface Ethernet ◦interface port-channel ◦interface vpc Before You Begin Create a named L3Out. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 interface type Specifies a port for the external interface. Example: apic1(config-leaf)# interface eth 1/20 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 211 Configuring Layer 3 External Connectivity Configuring Layer 3 Interfaces for a Named L3Out Step 4 Command or Action Purpose no switchport Configures the interface as a layer 3 interface, exposing the layer 3 commands in the configuration options. Example: apic1(config-leaf-if)# no switchport Step 5 vrf member tenant tenant-name vrf vrf-name l3out l3out-name Attaches the interface to the tenant VRF. Example: apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1 Step 6 [no] {ip | ipv6} address ip-prefix/masklen Configures IP addresses on the interface. The specified address can be declared as either: [eui64] [secondary] [preferred] Example: • preferred—The default source address for traffic from the interface. apic1(config-leaf-if)# ip address 10.1.1.1/24 apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred • secondary—The secondary address of the interface. With the optional eui64 keyword, the host can assign itself a 64-bit Extended Unique Identifier (EUI). In this mode, you can also configure ipv6 link-local, mac address, mtu, and other layer 3 properties on the interface. Examples This example shows how to assign a layer 3 port to a named L3Out. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/20 apic1(config-leaf-if)# no switchport apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1 apic1(config-leaf-if)# ip address 10.1.1.1/24 apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred This example shows how to assign a layer 3 subinterface to a named L3Out. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/5 apic1(config-leaf-if)# no switchport apic1(config-leaf-if)# vlan-domain member d1 apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 1/5.1000 apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1 apic1(config-leaf-if)# ip address 10.1.1.1/24 apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 212 Configuring Layer 3 External Connectivity Configuring Route Maps for a Named L3Out This example shows how to assign a layer 3 SVI to a named L3Out. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# interface vlan 200 apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 apic1(config-leaf-if)# ip address 10.1.1.1/24 apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 1/4 apic1(config-leaf-if)# vlan-domain member d1 apic1(config-leaf-if)# switchport trunk allowed vlan 200 tenant t1 external-svi l3out out1 Configuring Route Maps for a Named L3Out • Route-maps are configured under the leaf, VRF mode. • The following route-maps are created for every named L3Out : ◦Export—Route-map for routes advertised out of a routing protocol enabled on the L3Out. By default, no routes are exported out until you explicitly enable them in the route-map through one or more of match bridge-domain, match prefix-list and match community-list statements. ◦Import—Route-map for routes imported into the routing protocol on the L3Out. By default, all routes are imported. You can control specific routes to be imported by using one or more match prefix-list or match community-list statements. ◦Shared—Route-map that contains the routes and the contract provider/consumer policy that will be used for leaking the routes from this VRF to any other VRF that has the contract association. These route-maps are created when you associate a leaf to the L3Out through the vrf context tenant tenant-name vrf vrf-name l3out l3out-name command. • The scope of the route-maps under the named L3Out is always global and is applicable on all nodes where the Named L3Out is deployed. • All commands under the route-map (such as match prefix-list,match community-list, match bridge-domain) are the same as the route-map configuration for the Basic Mode discussed in the previous sections. Before You Begin Create a named L3Out. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 213 Configuring Layer 3 External Connectivity Configuring Route Maps for a Named L3Out Step 2 Command or Action Purpose leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 [no] vrf context tenant tenant-name vrf vrf-name l3out l3out-name Configures a tenant VRF on the node. Example: apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 l3out out1 Step 4 [no] route-map name Example: Creates a route-map and enters route-map configuration. This will be the import route-map. apic1(config-leaf-vrf)# route-map out1_in Step 5 [no] ip prefix-list list-name permit prefix/masklen [le {32 | 128}] Creates a prefix-list under the route-map. Example: apic1(config-leaf-vrf-route-map)# ip prefix-list p1 permit 15.1.1.0/24 Step 6 [no] match prefix-list list-name Example: apic1(config-leaf-vrf-route-map)# match prefix-list p1 Step 7 exit Matches a prefix-list that has already been created and enters the match mode to configure the route-control profile for the prefix-list. Returns to route-map configuration mode. Example: apic1(config-leaf-vrf-route-map-match)# exit Step 8 exit Returns to leaf VRF configuration mode. Example: apic1(config-leaf-vrf-route-map)# exit Step 9 [no] route-map name Example: Creates a route-map and enters route-map configuration. This will be the export route-map. apic1(config-leaf-vrf)# route-map out1_out Step 10 [no] ip prefix-list list-name permit prefix/masklen [le {32 | 128}] Example: apic1(config-leaf-vrf-route-map)# ip prefix-list p2 permit 16.1.1.0/24 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 214 Creates a prefix-list under the route-map. Configuring Layer 3 External Connectivity Configuring Route Maps for a Named L3Out Step 11 Command or Action Purpose [no] match prefix-list list-name Matches a prefix-list that has already been created and enters the match mode to configure the route-control profile for the prefix-list. Example: apic1(config-leaf-vrf-route-map)# match prefix-list p2 Step 12 set tag name Sets the tag value. The name parameter is an unsigned integer. Example: apic1(config-leaf-vrf-route-map-match)# set tag 100 Step 13 Returns to route-map configuration mode. exit Example: apic1(config-leaf-vrf-route-map-match)# exit Step 14 [no] match bridge-domain list-name Matches a bridge domain in order to export its public subnets through the protocol. Example: apic1(config-leaf-vrf-route-map)# match bridge-domain bd1 Step 15 Returns to route-map configuration mode. exit Example: apic1(config-leaf-vrf-route-map-match)# exit Step 16 [no] route-map name Creates a route-map and enters route-map configuration. This will be the shared route-map. Example: apic1(config-leaf-vrf)# route-map out1_shared Step 17 [no] ip prefix-list list-name permit prefix/masklen [le {32 | 128}] Creates a prefix-list under the route-map. Example: apic1(config-leaf-vrf-route-map)# ip prefix-list p3 permit 16.10.1.0/24 Step 18 [no] match prefix-list list-name Example: apic1(config-leaf-vrf-route-map)# match prefix-list p3 Step 19 contract provider name Matches a prefix-list that has already been created and enters the match mode to configure the route-control profile for the prefix-list. Adds contract, required to leak routes (matching this prefix-list) from this VRF. Example: apic1(config-leaf-vrf-route-map-match)# contract provider default Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 215 Configuring Layer 3 External Connectivity Configuring Routing Protocols for a Named L3Out Examples This example shows how to configure route maps for a named L3Out. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 l3out out1 # CREATE IMPORT ROUTE-MAP apic1(config-leaf-vrf)# route-map out1_in apic1(config-leaf-vrf-route-map)# ip prefix-list p1 permit 15.1.1.0/24 apic1(config-leaf-vrf-route-map)# match prefix-list p1 apic1(config-leaf-vrf-route-map-match)# exit apic1(config-leaf-vrf-route-map)# exit # CREATE EXPORT ROUTE-MAP apic1(config-leaf-vrf)# route-map out1_out apic1(config-leaf-vrf-route-map)# ip prefix-list p2 permit 16.1.1.0/24 apic1(config-leaf-vrf-route-map)# match prefix-list p2 apic1(config-leaf-vrf-route-map-match)# set tag 100 apic1(config-leaf-vrf-route-map-match)# exit apic1(config-leaf-vrf-route-map)# match bridge-domain bd1 apic1(config-leaf-vrf-route-map-match)# exit # CREATE SHARED ROUTE-MAP apic1(config-leaf-vrf)# route-map out1_shared apic1(config-leaf-vrf-route-map)# ip prefix-list p3 permit 16.10.1.0/24 apic1(config-leaf-vrf-route-map)# match prefix-list p3 apic1(config-leaf-vrf-route-map-match)# contract provider default Configuring Routing Protocols for a Named L3Out Configuring BGP for a Named L3Out • All commands under the BGP neighbor with the exception of route-map are identical to those in the Basic Mode of L3Out configuration. The BGP template configuration and the inheritance of the template are identical to the Basic Mode. • In the Named Mode of L3Out configuration, the route-map is applied at the L3Out level. By associating a neighbor with an L3Out, the route-map is automatically applied on the protocols on the L3Out. For this reason, the route-map option is not applicable and is not available under the BGP Neighbor. For the same reason, the route-map option is not available for OSPF Area and the distribute-list EIGRP option is not available under the interface. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 216 Configuring Layer 3 External Connectivity Configuring Routing Protocols for a Named L3Out Step 2 Command or Action Purpose leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 router bgp asn-number Enters BGP policy configuration. Example: apic1(config-leaf)# router bgp 100 Step 4 vrf member tenant tenant-name vrf vrf-name Example: Specifies the VRF instance to associate with subsequent policy configuration mode commands. apic1(config-bgp)# vrf member tenant exampleCorp vrf v100 Step 5 neighbor ip-address[ /masklength ] l3out l3out-name Specifies the IP address of the neighbor. Example: apic1(config-leaf-bgp-vrf)# neighbor 192.0.2.229 l3out out1 Step 6 remote-as asn Specifies Autonomous System Number of the neighbor. Example: apic1(config-leaf-bgp-vrf-neighbor)# remote-as 300 Step 7 allow-self-as-count count The count can be 1 to 10. The default is 3. Example: apic1(config-leaf-bgp-vrf-neighbor)# allow-self-as-count 5 Step 8 update-source ethernet interface-range Example: Update the Source IP for BGP Packets to one of loopback, physical, sub-interface or SVI interfaces.. apic1(config-leaf-bgp-vrf-neighbor)# update-source ethernet 1/3 Examples This example shows how to configure BGP routing protocol for a named L3Out. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router bgp 100 apic1(config-bgp)# vrf member tenant apic1(config-leaf-bgp-vrf)# neighbor apic1(config-leaf-bgp-vrf-neighbor)# apic1(config-leaf-bgp-vrf-neighbor)# apic1(config-leaf-bgp-vrf-neighbor)# exampleCorp vrf v1 192.0.2.229 l3out out1 remote-as 300 allow-self-as-count 5 update-source ethernet 1/3 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 217 Configuring Layer 3 External Connectivity Configuring Routing Protocols for a Named L3Out Configuring OSPF for a Named L3Out All commands under the router ospf default command, with the exception of area area-id route-map map-name out , are identical to those in the Basic Mode of L3Out configuration. The OSPF commands under the interface and the OSPF template inherit commands are also identical to the Basic Mode. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 router ospf default Creates an OSPF routing process and enters OSPF policy configuration. Example: apic1(config-leaf)# router ospf default Step 4 vrf member tenant tenant-name vrf vrf-name Enables a VRF in the OSPF session. Example: apic1(config-leaf-ospf)# vrf member tenant exampleCorp vrf v100 Step 5 area area-id l3out l3out-name Enables OSPF in the L3Out. Example: apic1(config-leaf-ospf-vrf)# area 0.0.0.1 l3out out1 Step 6 area area-id loopback loopback-address Example: apic1(config-leaf-ospf-vrf)# area 0.0.0.1 loopback 192.0.20.11 Step 7 area area-id nssa [no-redistribution] [default-information-originate] Example: apic1(config-leaf-ospf-vrf)# area 0.0.0.1 nssa Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 218 When OSPF is used as a connectivity protocol for BGP, OSPF advertises the loopback address which is used as the source of the BGP session. Note that the loopback IP address and not the loopback ID is used. In this case, a BGP session relying on OSPF will use the same loopback IP address in its update-source command. Defines a not-so-stubby area (NSSA). Configuring Layer 3 External Connectivity Configuring Routing Protocols for a Named L3Out Step 8 Command or Action Purpose exit Returns to the OSPF configuration mode. Example: apic1(config-leaf-ospf-vrf)# exit Step 9 Returns to leaf configuration mode. exit Example: apic1(config-leaf-ospf)# exit Step 10 interface type Specifies a port for the external interface. Example: apic1(config-leaf)# interface eth 1/20 Step 11 vlan-domain member domain-name Example: apic1(config-leaf-if)# vlan-domain member dom1 Step 12 Assign a VLAN domain to the interface. The VLAN domain must have already been created using the vlan-domain command in the global configuration mode. Configures the interface as a layer 3 interface, exposing the layer 3 commands in the configuration options. no switchport Example: apic1(config-leaf-if)# no switchport Step 13 vrf member tenant tenant-name vrf vrf-name l3out l3out-name Attaches the interface to the tenant VRF. Example: apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1 Step 14 [no] {ip | ipv6} address ip-prefix/masklen [eui64] [secondary] [preferred] Configures IP addresses on the interface. The specified address can be declared as either: Example: • preferred—The default source address for traffic from the interface. apic1(config-leaf-if)# ip address 10.1.1.1/24 apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred • secondary—The secondary address of the interface. With the optional eui64 keyword, the host can assign itself a 64-bit Extended Unique Identifier (EUI). In this mode, you can also configure ipv6 link-local, mac address, mtu, and other layer 3 properties on the interface. Step 15 {ip | ipv6} router ospf default area area-id Creates an OSPF routing process and enters OSPF policy configuration. Example: apic1(config-leaf-if)# ip router ospf default area 0.0.0.1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 219 Configuring Layer 3 External Connectivity Configuring Routing Protocols for a Named L3Out Examples This example shows how to configure OSPF routing protocol for a named L3Out. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router ospf default apic1(config-leaf-ospf)# vrf member tenant exampleCorp vrf v1 apic1(config-leaf-ospf-vrf)# area 0.0.0.1 l3out out1 apic1(config-leaf-ospf-vrf)# area 0.0.0.1 loopback 192.0.20.11 apic1(config-leaf-ospf-vrf)# area 0.0.0.1 nssa apic1(config-leaf-ospf-vrf)# exit apic1(config-leaf-ospf)# exit apic1(config-leaf)# interface eth 1/20 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# no switchport apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1 apic1(config-leaf-if)# ip address 10.1.1.1/24 apic1(config-leaf-if)# ip router ospf default area 0.0.0.1 Configuring EIGRP for a Named L3Out All EIGRP commands under vrf mode and interface mode, with the exception of ip distribute-list, are identical to those in the Basic Mode of L3Out configuration. This includes the EIGRP template and inherit commands. The ip distribute-list commands are not applicable to the Named Mode of L3Out configuration, as the route-maps are defined at the L3Out level and by associating an interface with the L3Out, the route-map distribute-list is automatically associated. For this reason, ip distribute-list is not available in the CLI as a option. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 leaf node-id Specifies the leaf to be configured. Example: apic1(config)# leaf 101 Step 3 router eigrp default Enters EIGRP policy configuration. Example: apic1(config-leaf)# router eigrp default Step 4 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with subsequent configuration mode commands. Example: apic1(config-eigrp)# vrf member tenant exampleCorp vrf v100 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 220 Configuring Layer 3 External Connectivity Configuring Routing Protocols for a Named L3Out Step 5 Command or Action Purpose autonomous-system asn l3out l3out-name Enters Autonomous System configuration for EIGRP. Example: apic1(config-eigrp-vrf)# autonomous-system 500 l3out out1 Step 6 Returns to the EIGRP configuration mode. exit Example: apic1(config-eigrp-vrf)# exit Step 7 Returns to leaf configuration mode. exit Example: apic1(config-eigrp)# exit Step 8 interface type Specifies a port for the external interface. Example: apic1(config-leaf)# interface eth 1/5 Step 9 vlan-domain member domain-name Example: apic1(config-leaf-if)# vlan-domain member dom1 Step 10 Assign a VLAN domain to the interface. The VLAN domain must have already been created using the vlan-domain command in the global configuration mode. Configures the interface as a layer 3 interface, exposing the layer 3 commands in the configuration options. no switchport Example: apic1(config-leaf-if)# no switchport Step 11 vrf member tenant tenant-name vrf vrf-name Attaches the interface to the tenant VRF. l3out l3out-name Example: apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1 Step 12 [no] {ip | ipv6} address ip-prefix/masklen [eui64] [secondary] [preferred] Configures IP addresses on the interface. The specified address can be declared as either: Example: • preferred—The default source address for traffic from the interface. apic1(config-leaf-if)# ip address 10.1.1.1/24 apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred • secondary—The secondary address of the interface. With the optional eui64 keyword, the host can assign itself a 64-bit Extended Unique Identifier (EUI). In this mode, you can also configure ipv6 link-local, mac address, mtu, and other layer 3 properties on the interface. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 221 Configuring Layer 3 External Connectivity Configuring External-L3 EPGs for a Named L3Out Step 13 Command or Action Purpose {ip | ipv6} router eigrp default Sets EIGRP policies to default. Example: apic1(config-leaf-if)# ip router eigrp default Examples This example shows how to configure EIGRP routing protocol for a named L3Out. apic1# configure apic1(config)# leaf 101 apic1(config-leaf)# router eigrp default apic1(config-eigrp)# vrf member tenant exampleCorp vrf v1 apic1(config-eigrp-vrf)# autonomous-system 500 l3out out1 apic1(config-eigrp-vrf)# exit apic1(config-eigrp)# exit apic1(config-leaf)# interface eth 1/5 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# no switchport apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1 apic1(config-leaf-if)# ip address 10.1.1.1/24 apic1(config-leaf-if)# ip router eigrp default Configuring External-L3 EPGs for a Named L3Out External-L3 EPGs are classified under a tenant VRF. All commands under the config-tenant-l3ext-epg mode are identical to those in the Basic Mode of L3Out configuration with the following differences: • The VRF is automatically associated with the EPG. The L3Out associates with a VRF and the EPG associates with the L3Out. • The external-l3 epg command is not available under the leaf vrf context tenant tenant-name vrf vrf-name l3out l3out-name command, as this configuration is not applicable for Named L3Outs. The external-l3 epg is automatically deployed on the leaf, when the external-l3 epg is created within a named L3Out and a leaf is associated with the same L3Out through the vrf context tenant tenant-name vrf vrf-name l3out l3out-name command. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 222 Configuring Layer 3 External Connectivity Configuring HSRP Step 2 Command or Action Purpose tenant tenant-name Enters the tenant configuration mode. Example: apic1(config)# tenant exampleCorp Step 3 external-l3 epg epg-name l3out l3out-name Enters the external-l3 EPG configuration mode. Example: apic1(config-tenant)# external-l3 epg epg1 l3out out1 Step 4 match {ip | ipv6} ip-address/masklength Creates a rule to match a subnet. Example: apic1(config-tenant-l3ext-epg)# match ip 192.0.20.0/24 apic1(config-tenant-l3ext-epg)# match ipv6 2001::1/64 Step 5 contract consumer contract-name Specifies the consumer contract for the EPG. Example: apic1(config-tenant-l3ext-epg)# contract consumer cConsumer1 Step 6 contract provider contract-name Specifies the provider contract for the EPG. Example: apic1(config-tenant-l3ext-epg)# contract provider cProvider1 Examples This example shows how to configure an external layer 3 EPG for a named L3Out. apic1# configure apic1(config)# tenant exampleCorp apic1(config-tenant)# external-l3 epg epg1 l3out out1 apic1(config-tenant-l3ext-epg)# match ip 192.0.20.0/24 apic1(config-tenant-l3ext-epg)# match ipv6 2001::1/64 apic1(config-tenant-l3ext-epg)# contract consumer cConsumer1 apic1(config-tenant-l3ext-epg)# contract provider cProvider1 Configuring HSRP Configuring HSRP in Cisco APIC Using Inline Parameters in NX-OS Style CLI HSRP is enabled when the leaf switch is configured. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 223 Configuring Layer 3 External Connectivity Configuring HSRP in Cisco APIC Using Template and Policy in NX-OS Style CLI Before You Begin • The tenant and VRF configured. • VLAN pools must be configured with the appropriate VLAN range defined and the appropriate Layer 3 domain created and attached to the VLAN pool. • The Attach Entity Profile must also be associated with the Layer 3 domain. • The interface profile for the leaf switches must be configured as required. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 Configure HSRP by creating inline parameters. Example: apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/17 apic1(config-leaf-if)# hsrp version 1 apic1(config-leaf-if)# hsrp use-bia apic1(config-leaf-if)# hsrp delay minimum 30 apic1(config-leaf-if)# hsrp delay reload 30 apic1(config-leaf-if)# hsrp 10 ipv4 apic1(config-if-hsrp)# ip 182.16.1.2 apic1(config-if-hsrp)# ip 182.16.1.3 secondary apic1(config-if-hsrp)# ip 182.16.1.4 secondary apic1(config-if-hsrp)# mac-address 5000.1000.1060 apic1(config-if-hsrp)# timers 5 18 apic1(config-if-hsrp)# priority 100 apic1(config-if-hsrp)# preempt apic1(config-if-hsrp)# preempt delay minimum 60 apic1(config-if-hsrp)# preempt delay reload 60 apic1(config-if-hsrp)# preempt delay sync 60 apic1(config-if-hsrp)# authentication none apic1(config-if-hsrp)# authentication simple apic1(config-if-hsrp)# authentication md5 apic1(config-if-hsrp)# authentication-key <mypassword> apic1(config-if-hsrp)# authentication-key-timeout <timeout> Configuring HSRP in Cisco APIC Using Template and Policy in NX-OS Style CLI HSRP is enabled when the leaf switch is configured. Before You Begin • The tenant and VRF configured. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 224 Configuring Layer 3 External Connectivity Configuring HSRP in Cisco APIC Using Template and Policy in NX-OS Style CLI • VLAN pools must be configured with the appropriate VLAN range defined and the appropriate Layer 3 domain created and attached to the VLAN pool. • The Attach Entity Profile must also be associated with the Layer 3 domain. • The interface profile for the leaf switches must be configured as required. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 Configure HSRP policy templates. Example: apic1(config)# leaf 101 apic1(config-leaf)# template hsrp interface-policy hsrp-intfPol1 tenant t9 apic1(config-template-hsrp-if-pol)# hsrp use-bia apic1(config-template-hsrp-if-pol)# hsrp delay minimum 30 apic1(config-template-hsrp-if-pol)# hsrp delay reload 30 apic1(config)# leaf 101 apic1(config-leaf)# template hsrp group-policy hsrp-groupPol1 tenant t9 apic1(config-template-hsrp-group-pol)# timers 5 18 apic1(config-template-hsrp-group-pol)# priority 100 apic1(config-template-hsrp-group-pol)# preempt apic1(config-template-hsrp-group-pol)# preempt delay minimum 60 apic1(config-template-hsrp-group-pol)# preempt delay reload 60 apic1(config-template-hsrp-group-pol)# preempt delay sync 60 Step 3 Use the configured policy templates Example: apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/17 apic1(config-leaf-if)# hsrp version 1 apic1(config-leaf-if)# inherit hsrp interface-policy hsrp-intfPol1 apic1(config-leaf-if)# hsrp 10 ipv4 apic1(config-if-hsrp)# ip 182.16.1.2 apic1(config-if-hsrp)# ip 182.16.1.3 secondary apic1(config-if-hsrp)# ip 182.16.1.4 secondary apic1(config-if-hsrp)# mac-address 5000.1000.1060 apic1(config-if-hsrp)# inherit hsrp group-policy hsrp-groupPol1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 225 Configuring Layer 3 External Connectivity Cisco ACI GOLF Cisco ACI GOLF Cisco ACI GOLF The Cisco ACI GOLF feature (also known as Layer 3 EVPN Services for Fabric WAN) enables much more efficient and scalable ACI fabric WAN connectivity. It uses the BGP EVPN protocol over OSPF for WAN routers that are connected to spine switches. Figure 19: Cisco ACIGolf Topology All tenant WAN connections use a single session on the spine switches where the WAN routers are connected. This aggregation of tenant BGP sessions towards the Data Center Interconnect Gateway (DCIG) improves control plane scale by reducing the number of tenant BGP sessions and the amount of configuration required for all of them. The network is extended out using Layer 3 subinterfaces configured on spine fabric ports. Transit routing with shared services using GOLF is not supported. A Layer 3 external outside network (L3extOut) for EVPN physical connectivity for a spine switch is specified under the infra tenant, and includes the following: • LNodeP (l3extInstP is not required within the L3Out in Tenant Infra ) • A provider label for the L3extOut for EVPN in tenant infra. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 226 Configuring Layer 3 External Connectivity Cisco ACI GOLF • OSPF protocol policies • BGP protocol policies All regular tenants use the above-defined physical connectivity. The L3extOut defined in regular tenants only needs the following: • An l3extConsLbl consumer label that must be matched with the same provider label of an L3extOut for EVPN in the infra tenant. Label matching enables application EPGs in other tenants to consume the LNodeP external L3extOut EPG. • An l3extInstP with subnets and contracts. The scope of the subnet is used to control import/export route control and security policies. • The BGP EVPN session in the matching provider L3extOut in the infra tenant advertises the tenant routes defined in this L3extOut. Observe the following GOLF guidelines and limitations: • At this time, only a single GOLF provider policy can be deployed on spine switch interfaces for the whole fabric. • Up to APIC release 2.0(2), GOLF is not supported with multipod. In release 2.0 (2) the two features are supported in the same fabric only over Cisco Nexus N9000K switches without “EX” on the end of the switch name; for example, N9K-9312TX. Since the 2.1(1) release, the two features can be deployed together over all the switches used in the multipod and EVPN topologies. • When configuring GOLF on a spine switch, wait for the control plane to converge before configuring GOLF on another spine. • A spine switch can be added to multiple provider GOLF outside networks (GOLF Outs), but the provider labels have to be different for each GOLF Out. Also, in this case, the OSPF Area has to be different on each of the L3extOuts and use different loopback addresses. • The BGP EVPN session in the matching provider L3extOut in the infra tenant advertises the tenant routes defined in this L3extOut. • When deploying three GOLF Outs, if only 1 has a provider/consumer label for GOLF, and 0/0 export aggregation, APIC will export all routes. This is the same as existing L3extOut on leaf switches for tenants. • If there is direct peering between a spine switch and a data center interconnect (DCI) router, the transit routes from leaf switches to the ASR have the next hop as the PTEP of the leaf. In this case, define a static route on the ASR for the TEP range of that ACI pod. Also, if the DCI is dual-homed to the same pod, then the precedence (administrative distance) of the static route should be the same as the route received through the other link. • The default bgpPeerPfxPol policy restricts routes to 20, 000. For ACI WAN Interconnect peers, increase this as needed. • In a deployment scenario where there are two L3extOuts on one spine, and one of them has the provider label prov1 and peers with the DCI 1, the second L3extOut peers with DCI 2 with provider label prov2. If the tenant VRF has a consumer label pointing to any 1 of the provider labels (either prov1 or prov2), the tenant route will be sent out both DCI 1 and DCI 2. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 227 Configuring Layer 3 External Connectivity Configuration Tasks to Configure Cisco ACI GOLF Services Using the NX-OS Style CLI Note When you configure Layer 3 Outside (L3Out) connections to external routers, or multipod connections through an Inter-Pod Network (IPN), it is critical that the MTU be set appropriately on both sides. On some platforms, such as ACI, Cisco NX-OS, and Cisco IOS, the configurable MTU value takes into account packet headers (resulting in a max packet size to be set as 9000 bytes), whereas other platforms such as IOS-XR configure the MTU value exclusive of packet headers (resulting in a max packet size of 8986 bytes). For the appropriate MTU values for each platform, see the relevant configuration guides. Cisco highly recommends you test the MTU using CLI-based commands. For example, on the Cisco NX-OS CLI, use a command such as ping 1.1.1.1 df-bit packet-size 9000 source-interface ethernet 1/1. Configuration Tasks to Configure Cisco ACI GOLF Services Using the NX-OS Style CLI Perform the following tasks to configure GOLF services (using the BGP EVPN protocol), with the NX-OS style CLI: • Configure the infra tenant for BGP EVPN, including the VLAN domain, VRF, Interface IP addressing, and OSPF. • Configure BGP on the spine node to support BGP EVPN. • Configure a tenant for BGP EVPN. • Configure the BGP EVPN route target, route map, and prefix-epg for the tenant. • Configure BGP address-families to enable distributing BGP EVPN type-2 (MAC-IP) host routes to the DCIG, with the host-rt-enable command . Configuring a Spine and the Infra Tenant for BGP EVPN, Using the NX-OS Style CLI This task describes how to configure the infra tenant for BGP EVPN, including the VLAN domain, VRF, Interface IP addressing, and OSPF in the following steps: Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. apic1# configure Step 2 vlan-domain vlan-domain-name Creates a VLAN domain. dynamic apic1(config)# vlan-domain Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 228 evpn-dom dynamic Configuring Layer 3 External Connectivity Configuring a Spine and the Infra Tenant for BGP EVPN, Using the NX-OS Style CLI Step 3 Command or Action Purpose spine spine-name Creates the spine or enters spine configuration mode. apic1(config)# spine 111 Step 4 vrf context tenanttenant-name Associates the VRF with the tenant. vrf vrf-name apic1(config-spine)# vrf context tenant infra vrf overlay-1 Step 5 router-id A.B.C.D Configures the router ID for the VRF. apic1(config-spine-vrf)# router-id 10.10.3.3 Step 6 Returns to spine configuration mode. exit apic1(config-spine-vrf)# exit Step 7 interface ethernet slot/port Configures an interface for a spine node. apic1(config-spine)# interface ethernet 1/33 Step 8 Step 9 vlan-domain member vlan-domain-name Associates the interface with the VLAN domain. exit Returns to spine configuration mode. apic1(config-spine-if)# vlan-domain member evpn-dom apic1(config-spine-if)# exit Step 10 Step 11 Creates a sub-interface. interface ethernet sub-interface-id apic1(config-spine)# interface ethernet 1/33.4 vrf member tenanttenant-name Associates the interface with the overlay-1 VRF and the infra tenant. vrf vrf-name apic1(config-spine-if)# vrf member tenant infra vrf overlay-1 Step 12 mtu mtu-value Sets the maximum transmission unit (MTU) for the interface. apic1(config-spine-if)# mtu 1500 Step 13 ip address A.B.C.D/LEN Sets the IP address for the interface. apic1(config-spine-if)# ip address 5.0.0.1/24 Step 14 Step 15 ip router ospf default areaospf-area-id Sets the default OSPF area ID for the interface. exit Returns to spine configuration mode. apic1(config-spine-if)# ip router ospf default area 0.0.0.150 apic1(config-spine-if)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 229 Configuring Layer 3 External Connectivity Configuring a Spine and the Infra Tenant for BGP EVPN, Using the NX-OS Style CLI Step 16 Command or Action Purpose interface ethernet slot/port Configures an interface for a spine node. apic1(config-spine)# interface ethernet 1/34 Step 17 Step 18 vlan-domain member vlan-domain-name Associates the interface with the VLAN domain. exit Returns to spine configuration mode. apic1(config-spine-if)# vlan-domain member evpn-dom apic1(config-spine-if)# exit Step 19 Step 20 interface ethernet sub-interface-id Creates a sub-interface. apic1(config-spine)# interface ethernet 1/34.4 vrf member tenanttenant-name Associates the interface with the overlay-1 VRF and the infra tenant. vrf vrf-name apic1(config-spine-if)# vrf member tenant infra vrf overlay-1 Step 21 mtu mtu-value Sets the maximum transmission unit (MTU) for the interface. apic1(config-spine-if)# mtu 1500 Step 22 ip address A.B.C.D/LEN Sets the IP address for the interface. apic1(config-spine-if)# ip address 2.0.0.1/24 Step 23 Step 24 ip router ospf default areaospf-area-id Sets the default OSPF area ID for the interface. exit Returns to spine configuration mode. apic1(config-spine-if)# ip router ospf default area 0.0.0.200 apic1(config-spine-if)# exit Step 25 router ospf default Configures OSPF for the spine. apic1(config-spine)# router ospf default Step 26 vrf member tenant tenant-name vrf vrf-name Associates the Router OSPF policy with the overlay-1 VRF and infra tenant. apic1(config-spine-ospf)# vrf member tenant infra vrf overlay-1 Step 27 Step 28 area area-id loopback loopback-ip-address Configure an OSPF area for the OSPF policy. area area-id loopback loopback-ip-address Configure another OSPF area for the OSPF policy. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 230 apic1(config-spine-ospf-vrf)# area 0.0.0.150 loopback 10.10.5.3 apic1(config-spine-ospf-vrf)# area 0.0.0.200 loopback 10.10.4.3 Configuring Layer 3 External Connectivity Configuring BGP to Support BGP EVPN on a Spine, Using the NX-OS Style CLI Step 29 Command or Action Purpose exit Returns to spine OSPF configuration mode. apic1(config-spine-ospf-vrf)# exit Step 30 Returns to spine configuration mode. exit apic1(config-spine-ospf)# exit Configuring BGP to Support BGP EVPN on a Spine, Using the NX-OS Style CLI This task shows how to configure BGP on the spine to support BGP EVPN in the following steps: Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. apic1# configure Step 2 spine spine-name Creates the spine or enters spine configuration mode. apic1(config)# spine 111 Step 3 router bgp AS-number Configures BGP for the spine node. apic1(config-spine)# router bgp 100 Step 4 vrf context tenanttenant-name Associates the Router BGP policy with the infra tenant and the overlay-1 VRF. vrf vrf-name apic1(config-spine-bgp)# vrf context tenant infra vrf overlay-1 Step 5 vrf context tenanttenant-name Associates the Router BGP policy with the infra tenant and the overlay-1 VRF. vrf vrf-name apic1(config-spine-bgp-vrf)# vrf context tenant infra vrf overlay-1 Step 6 neighbor neighbor-ip-address Configures the IP address for an EVPN BGP neighbor. evpn apic1(config-spine-bgp-vrf)# neighbor 10.10.4.1 evpn Step 7 label label-name Assigns a label to the neighbor. apic1(config-spine-bgp-vrf-neighbor)# label evpn-aci Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 231 Configuring Layer 3 External Connectivity Configuring a Tenant for BGP EVPN Using the NX-OS Style CLI Command or Action Step 8 Purpose update-source Sets the update source to be the neighbor loopback IP loopbackloopback-ip-address address. vrf vrf-name apic1(config-spine-bgp-vrf-neighbor)# update-source loopback 10.10.4.3 Step 9 remote-as AS-number Specifies the autonomous system (AS) number of the neighbor. The valid value can be from 1 to 4294967295. apic1(config-spine-bgp-vrf-neighbor)# remote-as 100 Step 10 exit Returns to BGP VRF configuration mode. apic1(config-spine-bgp-vrf-neighbor)# exit Step 11 neighbor neighbor-ip-address Configures the IP address for an EVPN BGP neighbor. evpn apic1(config-spine-bgp-vrf)# neighbor 10.10.5.1 evpn Step 12 label label-name Assigns a label to the neighbor. apic1(config-spine-bgp-vrf-neighbor)# label evpn-aci2 Step 13 update-source Sets the update source to be the neighbor loopback IP loopbackloopback-ip-address address. vrf vrf-name apic1(config-spine-bgp-vrf-neighbor)# update-source loopback 10.10.5.3 Step 14 remote-as AS-number Specifies the autonomous system (AS) number of the neighbor. The valid value can be from 1 to 4294967295. apic1(config-spine-bgp-vrf-neighbor)# remote-as 100 Configuring a Tenant for BGP EVPN Using the NX-OS Style CLI This task shows how to configure a tenant for BGP EVPN in the following steps: Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 232 Configuring Layer 3 External Connectivity Configuring a Tenant for BGP EVPN Using the NX-OS Style CLI Step 2 Command or Action Purpose tenant tenant-name Creates the tenant or enters tenant configuration mode. apic1(config)# tenant sky Step 3 vrf context vrf-name Creates a VRF for the tenant. apic1(config-tenant)# vrf context vrf-sky Step 4 Returns to tenant configuration mode. exit apic1(config-tenant-vrf)# exit Step 5 bridge-domain bd-name Creates a bridge domain apic1(config-tenant)# bridge-domain bd-sky Step 6 vrf member vrf-name Associates the bridge domain with the VRF and tenant. apic1(config-tenant-bd)# vrf member vrf-sky Step 7 Returns to tenant configuration mode. exit apic1(config-tenant-bd)# exit Step 8 Step 9 interface bridge-domain bd-name Creates an interface for a bridge domain. ip address A.B.C.D/LEN Assigns an IP address and length to the bridge-domain interface. apic1(config-tenant)# interface bridge-domain bd_sky apic1(config-tenant-interface)# ip address 59.10.1.1/24 Step 10 Returns to tenant configuration mode. exit apic1(config-tenant-interface)# exit Step 11 bridge-domain bd-name Creates a bridge domain apic1(config-tenant)# bridge-domain bd-sky2 Step 12 vrf member vrf-name Associates the bridge domain with the VRF and tenant. apic1(config-tenant-bd)# vrf member vrf-sky Step 13 Returns to tenant configuration mode. exit apic1(config-tenant-bd)# exit Step 14 Step 15 interface bridge-domain bd-name Creates an interface for a bridge domain. ip address A.B.C.D/LEN Assigns an IP address and length to the bridge-domain interface. apic1(config-tenant)# interface bridge-domain bd_sky2 apic1(config-tenant-interface)# ip address 59.11.1.1/24 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 233 Configuring Layer 3 External Connectivity Configuring a Route Map Step 16 Command or Action Purpose exit Returns to tenant configuration mode. apic1(config-tenant-interface)# exit Configuring a Route Map This task shows how to configure a route map to advertise bridge-domain subnets through BGP EVPN. Each bridge domain is advertised through a different BGP EVPN session on the spine, with a unique provider label. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. apic1# configure Step 2 spine spine-name Creates a spine or enters spine configuration mode. apic1(config)# spine 111 Step 3 vrf context tenanttenant-name Enters creates a VRF or enters VRF configuration mode. vrf vrf-name apic1(config-spine)# vrf context tenant sky vrf vrf_sky Step 4 address-family { ipv4 | ipv6 } Sets IPv4 or IPv6 unicast address family for the VRF. unicast apic1(config-spine-vrf)# address-family ipv4 unicast Step 5 Assigns an export route target to the address family. route-target mode extended-community-number apic1(config-spine-vrf-af)# route-target export 100:1 Step 6 Assigns an import route target to the address family. route-target mode extended-community-number apic1(config-spine-vrf-af)# route-target import 100:1 Step 7 exit Returns to spine VRF configuration mode. apic1(config-spine-vrf-af)# exit Step 8 route-map route-map-name Creates a route map for EVPN (with prefix learned from a transit network). apic1(config-spine-vrf)# route map rmap Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 234 Configuring Layer 3 External Connectivity Configuring a Route Map Step 9 Command or Action Purpose ip prefix-list ip-pl-name permit A.B.C.D/LEN Adds an IP prefix list to the route map to permit traffic from the specified subnet. apic1(config-spine-vrf-route-map)# ip prefix-list pl permit 11.10.10.0/24 Step 10 match bridge-domain bd-name Configures the route-map to match traffic belonging to the bridge domain. apic1(config-spine-vrf-route-map)# match bridge-domain bd_sky Step 11 Returns to spine VRF route-map configuration mode. exit apic1(config-spine-vrf-route-map-match)# exit Step 12 match prefix-list pl-name Sets the route-map to match the specified prefix-list. apic1(config-spine-vrf-route-map)# match prefix-list pl Step 13 Returns to spine VRF route-map configuration mode. exit apic1(config-spine-vrf-route-map-match)# exit Step 14 Returns to spine VRF configuration mode. exit apic1(config-spine-vrf-route-map)# exit Step 15 Step 16 evpn export maproute-map-name label consumer-label-name Assigns a consumer label to the VRF. route-map route-map-name Creates a route map for EVPN (with prefix learned from a transit network). apic1(config-spine-vrf)# evpn export map rmap label evpn-aci apic1(config-spine-vrf)# route map rmap2 Step 17 match bridge-domain bd-name Configures the route-map to match traffic belonging to the bridge domain. apic1(config-spine-vrf-route-map)# match bridge-domain bd_sky Step 18 Returns to spine VRF route-map configuration mode. exit apic1(config-spine-vrf-route-map-match)# exit Step 19 match prefix-list pl-name Sets the route-map to match the specified prefix-list. apic1(config-spine-vrf-route-map)# match prefix-list pl Step 20 exit Returns to spine VRF route-map configuration mode. apic1(config-spine-vrf-route-map-match)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 235 Configuring Layer 3 External Connectivity Enabling Distributing BGP EVPN Type-2 Host Routes to a DCIG Using the NX-OS Style CLI Step 21 Command or Action Purpose exit Returns to spine VRF configuration mode. apic1(config-spine-vrf-route-map)# exit Step 22 evpn export maproute-map-name label consumer-label-name Assigns a consumer label to the VRF. Step 23 external-l3 epgepg-name apic1(config-spine-vrf)# external-l3 epg l3_sky Step 24 vrf membervrf-name apic1(config-spine-vrf-l3ext-epg)# vrf member vrf_sky Step 25 match ipA.B.C.D/LEN Configure the subnet that identifies hosts as being part of the EPG. apic1(config-spine-vrf)# evpn export map rmap label evpn-aci2 apic1(config-spine-vrf-l3ext-epg)# match ip 80.10.1.0/24 Enabling Distributing BGP EVPN Type-2 Host Routes to a DCIG Using the NX-OS Style CLI Procedure Command or Action Purpose Step 1 Configure distributing EVPN type-2 host routes to a DCIG This template will be available on all nodes with the following commands in the BGP address family where tenant bgp_t1 has a VRF configuration mode. deployment. To disable distributing EVPN type-2 host routes, enter the no host-rt-enable command. Example: apic1(config)# leaf 101 apic1(config-leaf)# template bgp address-family bgpAf1 tenant bgp_t1 apic1(config-bgp-af)# distance 250 240 230 apic1(config-bgp-af)# host-rt-enable apic1(config-bgp-af)# exit Cisco ACI GOLF Configuration Example, Using the NX-OS Style CLI These examples show the CLI commands to configure GOLF Services, which uses the BGP EVPN protocol over OSPF for WAN routers that are connected to spine switches. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 236 Configuring Layer 3 External Connectivity Cisco ACI GOLF Configuration Example, Using the NX-OS Style CLI Configuring the infra Tenant for BGP EVPN The following example shows how to configure the infra tenant for BGP EVPN, including the VLAN domain, VRF, Interface IP addressing, and OSPF: configure vlan-domain evpn-dom dynamic exit spine 111 # Configure Tenant Infra VRF overlay-1 on the spine. vrf context tenant infra vrf overlay-1 router-id 10.10.3.3 exit interface ethernet 1/33 vlan-domain member golf_dom exit interface ethernet 1/33.4 vrf member tenant infra vrf overlay-1 mtu 1500 ip address 5.0.0.1/24 ip router ospf default area 0.0.0.150 exit interface ethernet 1/34 vlan-domain member golf_dom exit interface ethernet 1/34.4 vrf member tenant infra vrf overlay-1 mtu 1500 ip address 2.0.0.1/24 ip router ospf default area 0.0.0.200 exit router ospf default vrf member tenant infra vrf overlay-1 area 0.0.0.150 loopback 10.10.5.3 area 0.0.0.200 loopback 10.10.4.3 exit exit Configuring BGP on the Spine Node The following example shows how to configure BGP to support BGP EVPN: Configure spine 111 router bgp 100 vrf member tenant infra vrf overlay- 1 neighbor 10.10.4.1 evpn label golf_aci update-source loopback 10.10.4.3 remote-as 100 exit neighbor 10.10.5.1 evpn label golf_aci2 update-source loopback 10.10.5.3 remote-as 100 exit exit exit Configuring a Tenant for BGP EVPN The following example shows how to configure a tenant for BGP EVPN, including a gateway subnet which will be advertised through a BGP EVPN session: configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 237 Configuring Layer 3 External Connectivity Troubleshooting EVPN Type-2 Route Distribution to a DCIG tenant sky vrf context vrf_sky exit bridge-domain bd_sky vrf member vrf_sky exit interface bridge-domain bd_sky ip address 59.10.1.1/24 exit bridge-domain bd_sky2 vrf member vrf_sky exit interface bridge-domain bd_sky2 ip address 59.11.1.1/24 exit exit Configuring the BGP EVPN Route Target, Route Map, and Prefix EPG for the Tenant The following example shows how to configure a route map to advertise bridge-domain subnets through BGP EVPN. configure spine 111 vrf context tenant sky vrf vrf_sky address-family ipv4 unicast route-target export 100:1 route-target import 100:1 exit route-map rmap ip prefix-list p1 permit 11.10.10.0/24 match bridge-domain bd_sky exit match prefix-list p1 exit evpn export map rmap label golf_aci route-map rmap2 match bridge-domain bd_sky exit match prefix-list p1 exit exit evpn export map rmap label golf_aci2 external-l3 epg l3_sky vrf member vrf_sky match ip 80.10.1.0/24 exit Troubleshooting EVPN Type-2 Route Distribution to a DCIG For optimal traffic forwarding in an EVPN topology, you can enable fabric spines to distribute host routes to a Data Center Interconnect Gateway (DCIG) using EVPN type-2 (MAC-IP) routes along with the public BD subnets in the form of BGP EVPN type-5 (IP Prefix) routes. This is enabled using the HostLeak object. If you encounter problems with route distribution, use the steps in this topic to troubleshoot. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 238 Configuring Layer 3 External Connectivity Troubleshooting EVPN Type-2 Route Distribution to a DCIG Procedure Step 1 Verify that HostLeak object is enabled under the VRF-AF in question, by entering a command such as the following in the spine-switch CLI: Example: spine1# ls /mit/sys/bgp/inst/dom-apple/af-ipv4-ucast/ ctrl-l2vpn-evpn ctrl-vpnv4-ucast hostleak summary Step 2 Verify that the config-MO has been successfully processed by BGP, by entering a command such as the following in the spine-switch CLI: Example: spine1# show bgp process vrf apple Look for output similar to the following: Information for address family IPv4 Unicast in VRF apple Table Id : 0 Table state : UP Table refcount : 3 Peers Active-peers Routes Paths Networks 0 0 0 0 0 Aggregates 0 Redistribution None Wait for IGP convergence is not configured GOLF EVPN MAC-IP route is enabled EVPN network next-hop 192.41.1.1 EVPN network route-map map_pfxleakctrl_v4 Import route-map rtctrlmap-apple-v4 EVPN import route-map rtctrlmap-evpn-apple-v4 Step 3 Verify that the public BD-subnet has been advertised to DCIG as an EVPN type-5 route: Example: spine1# show bgp l2vpn evpn 10.6.0.0 vrf overlay-1 Route Distinguisher: 192.41.1.5:4123 (L3VNI 2097154) BGP routing table entry for [5]:[0]:[0]:[16]:[10.6.0.0]:[0.0.0.0]/224, version 2088 Paths: (1 available, best #1) Flags: (0x000002 00000000) on xmit-list, is not in rib/evpn Multipath: eBGP iBGP Advertised path-id 1 Path type: local 0x4000008c 0x0 ref 1, path is valid, is best path AS-Path: NONE, path locally originated 192.41.1.1 (metric 0) from 0.0.0.0 (192.41.1.5) Origin IGP, MED not set, localpref 100, weight 32768 Received label 2097154 Community: 1234:444 Extcommunity: RT:1234:5101 4BYTEAS-GENERIC:T:1234:444 Path-id 1 advertised to peers: 50.41.50.1 In the Path type entry, ref 1 indicates that one route was sent. Step 4 Verify whether the host route advertised to the EVPN peer was an EVPN type-2 MAC-IP route: Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 239 Configuring Layer 3 External Connectivity Troubleshooting EVPN Type-2 Route Distribution to a DCIG Example: spine1# show bgp l2vpn evpn 10.6.41.1 vrf overlay-1 Route Distinguisher: 10.10.41.2:100 (L2VNI 100) BGP routing table entry for [2]:[0]:[2097154]:[48]:[0200.0000.0002]:[32]:[10.6.41 .1]/272, version 1146 Shared RD: 192.41.1.5:4123 (L3VNI 2097154) Paths: (1 available, best #1) Flags: (0x00010a 00000000) on xmit-list, is not in rib/evpn Multipath: eBGP iBGP Advertised path-id 1 Path type: local 0x4000008c 0x0 ref 0, path is valid, is best path AS-Path: NONE, path locally originated EVPN network: [5]:[0]:[0]:[16]:[10.6.0.0]:[0.0.0.0] (VRF apple) 10.10.41.2 (metric 0) from 0.0.0.0 (192.41.1.5) Origin IGP, MED not set, localpref 100, weight 32768 Received label 2097154 2097154 Extcommunity: RT:1234:16777216 Path-id 1 advertised to peers: 50.41.50.1 The Shared RD line indicates the RD/VNI shared by the EVPN type-2 route and the BD subnet. The EVPN Network line shows the EVPN type-5 route of the BD-Subnet. The Path-id advertised to peers indicates the path advertised to EVPN peers. Step 5 Verify that the EVPN peer (a DCIG) received the correct type-2 MAC-IP route and the host route was successfully imported into the given VRF, by entering a command such as the following on the DCIG device (assuming that the DCIG is a Cisco ASR 9000 switch in the example below): Example: RP/0/RSP0/CPU0:asr9k#show bgp vrf apple-2887482362-8-1 10.6.41.1 Tue Sep 6 23:38:50.034 UTC BGP routing table entry for 10.6.41.1/32, Route Distinguisher: 44.55.66.77:51 Versions: Process bRIB/RIB SendTblVer Speaker 2088 2088 Last Modified: Feb 21 08:30:36.850 for 28w2d Paths: (1 available, best #1) Not advertised to any peer Path #1: Received by speaker 0 Not advertised to any peer Local 192.41.1.1 (metric 42) from 10.10.41.1 (192.41.1.5) Received Label 2097154 Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported Received Path ID 0, Local Path ID 1, version 2088 Community: 1234:444 Extended community: 0x0204:1234:444 Encapsulation Type:8 Router MAC:0200.c029.0101 RT:1234:5101 RIB RNH: table_id 0xe0000190, Encap 8, VNI 2097154, MAC Address: 0200.c029.0101, IP Address: 192.41.1.1, IP table_id 0x00000000 Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 192.41.1.5:4123 In this output, the received RD, next hop, and attributes are the same for the type-2 route and the BD subnet. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 240 Configuring Layer 3 External Connectivity Multipod_Fabric Multipod_Fabric About Multipod Fabric Multipod enables provisioning a more fault tolerant fabric comprised of multiple pods with isolated control plane protocols. Also, multipod provides more flexibility with regard to the full mesh cabling between leaf and spine switches. For example, if leaf switches are spread across different floors or different buildings, multipod enables provisioning multiple pods per floor or building and providing connectivity between pods through spine switches. Multipod uses MP-BGP EVPN as the control-plane communication protocol between the ACI spines in different Pods. WAN routers can be provisioned in the IPN, directly connected to spine switches or connected to border leaf switches. Multipod uses a single APIC cluster for all the pods; all the pods act as a single fabric. Individual APIC controllers are placed across the pods but they are all part of a single APIC cluster. Assigning Switches in a Multipod Fabric Before You Begin The node group and L3Out policies have already been created. Procedure Step 1 Command or Action Purpose configure Enter global configuration mode. Example: apic1# configure Step 2 [no] system switch-id serial-number switch-id For each switch in the multipod fabric, declare switch-name [pod pod-id] [role {leaf | spine}] the associated pod and the role (leaf or spine) of the switch. Repeat this command for each leaf and spine switch in the multipod fabric. Example: apic1(config)# system switch-id SAL1748H56D 201 ifav4-spine1 pod 1 role spine Step 3 [no] system pod pod-id tep-pool ip-prefix/length Configure a tunnel endpoint IP address pool for a pod. Repeat this command for each pod in the multipod fabric. Example: apic1(config)# system pod 1 tep-pool 10.0.0.0/16 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 241 Configuring Layer 3 External Connectivity Configuring Fabric-External Connectivity for a Multipod Fabric This example shows how to assign spine and leaf switches in a two-pod fabric. apic1# configure apic1(config)# system apic1(config)# system apic1(config)# system apic1(config)# system apic1(config)# system apic1(config)# system apic1(config)# system apic1(config)# system apic1(config)# system apic1(config)# system apic1(config)# system apic1(config)# system switch-id SAL1748H56D 201 ifav4-spine1 pod 1 role spine switch-id SAL1938P7A6 202 ifav4-spine3 pod 1 role spine switch-id SAL1819RXP4 101 ifav4-leaf1 pod 1 role leaf switch-id SAL1803L25H 102 ifav4-leaf2 pod 1 role leaf switch-id SAL1934MNY0 103 ifav4-leaf3 pod 1 role leaf switch-id SAL1934MNY3 104 ifav4-leaf4 pod 1 role leaf switch-id SAL1931LA3B 203 ifav4-spine2 pod 2 role spine switch-id FGE173400A9 204 ifav4-spine4 pod 2 role spine switch-id SAL1938PHBB 105 ifav4-leaf5 pod 2 role leaf switch-id SAL1942R857 106 ifav4-leaf6 pod 2 role leaf pod 1 tep-pool 10.0.0.0/16 pod 2 tep-pool 10.1.0.0/16 What to Do Next Configure fabric-external connectivity. Configuring Fabric-External Connectivity for a Multipod Fabric Before You Begin • The node group and L3Out policies have already been created. • Switches have been assigned to pods. Procedure Step 1 Command or Action Purpose configure Enter global configuration mode. Example: apic1# configure Step 2 [no] fabric-external controller-number Example: apic1(config)# fabric-external 1 Step 3 [no] bgp evpn peering [password Configure BGP EVPN peering profile. You can configure a peering password, peering-password] [type {automatic_with_full_mesh | automatic_with_rr}] and you can set the type to be either full mesh or with route-reflector. Example: apic1(config-fabric-external)# bgp evpn peering Step 4 [no] pod pod-id Example: apic1(config-fabric-external)# pod 1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 242 Select a pod for configuring. Configuring Layer 3 External Connectivity Configuring Fabric-External Connectivity for a Multipod Fabric Command or Action Step 5 Purpose [no] interpod data hardware-proxy ip-addr/mask Configure the anycast hardware-proxy IP address for each pod for inter-pod traffic. Example: apic1(config-fabric-external-pod)# interpod data hardware-proxy 100.11.1.1/32 Step 6 [no] bgp evpn peering [password peering-password] [type {automatic_with_full_mesh | automatic_with_rr}] Example: apic1(config-fabric-external-pod)# bgp evpn peering Step 7 Return to BGP EVPN peering profile configuration. exit Example: apic1(config-fabric-external-pod)# exit Step 8 Repeat steps 4 through 7 for each pod in the multipod fabric. Step 9 [no] route-map interpod-import Example: apic1(config-fabric-external)# route-map interpod-import Step 10 Configure a route-map that contains subnets on the inter-pod network (IPN) that will be allowed into the fabric through the OSPF protocol. [no] ip prefix-list prefix-list-name [permit ip-address/len Example: apic1(config-fabric-external-route-map)# ip prefix-list default permit 0.0.0.0/0 Step 11 Return to fabric-external configuration mode. exit Example: apic1(config-fabric-external-route-map)# exit Step 12 [no] route-target extended ASN4:NN Example: apic1(config-fabric-external)# route-target extended 5:16 Step 13 Route targets are carried as extended community attributes. Enter the community number in the AA4:NN2 format: 1-4294967295: 1-65535. exit This example shows how to configure fabric-external connectivity for a multipod fabric. apic1# configure apic1(config)# fabric-external 1 apic1(config-fabric-external)# bgp evpn peering apic1(config-fabric-external)# pod 1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 243 Configuring Layer 3 External Connectivity Configuring Spine Interfaces and OSPF for a Multipod Fabric apic1(config-fabric-external-pod)# interpod data hardware-proxy 100.11.1.1/32 apic1(config-fabric-external-pod)# bgp evpn peering apic1(config-fabric-external-pod)# exit apic1(config-fabric-external)# pod 2 apic1(config-fabric-external-pod)# interpod data hardware-proxy 200.11.1.1/32 apic1(config-fabric-external-pod)# bgp evpn peering apic1(config-fabric-external-pod)# exit apic1(config-fabric-external)# route-map interpod-import apic1(config-fabric-external-route-map)# ip prefix-list default permit 0.0.0.0/0 apic1(config-fabric-external-route-map)# exit apic1(config-fabric-external)# route-target extended 5:16 apic1(config-fabric-external)# exit What to Do Next Configure spine interfaces and OSPF. Configuring Spine Interfaces and OSPF for a Multipod Fabric Before You Begin • Switches have been assigned to pods. • A VLAN domain must exist. Procedure Step 1 Command or Action Purpose configure Enter global configuration mode. Example: apic1# configure Step 2 spine spine-id Example: You can specify the spine switch by an ID number in the range of 101 to 4000 or by name, such as 'spine1.' apic1(config)# spine 104 Step 3 [no] vrf context tenant infra vrf vrf-name Example: apic1(config-spine)# vrf context tenant infra vrf overlay-1 Step 4 [no] router-id A.B.C.D Configure a router identifier (ID). Example: apic1(config-spine-vrf)# router-id 201.201.201.201 Step 5 exit Example: apic1(config-spine-vrf)# exit Step 6 [no] interface ethernet slot/port Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 244 Return to spine configuration mode. Configuring Layer 3 External Connectivity Configuring Spine Interfaces and OSPF for a Multipod Fabric Command or Action Purpose Example: apic1(config-spine)# interface ethernet 1/1 Step 7 [no] vlan-domain member domain-name apic1(config-spine)# vlan-domain member l3Dom The VLAN domain must already exist, having been created using the vlan-domain domain-name command in the global configuration mode. exit Return to spine configuration mode. Example: Step 8 Example: apic1(config-spine-if)# exit Step 9 [no] interface ethernet type/slot.subinterface Encapsulation for the subinterface must be 4. Example: apic1(config-spine)# interface ethernet 1/1.4 Step 10 [no] vrf member tenant infra vrf vrf-name Configure the interface as a member of the tenant VRF. Example: apic1(config-spine-if)# vrf member tenant infra vrf overlay-1 Step 11 [no] ip address ip-address Example: apic1(config-spine-if)# ip address 201.1.1.1/30 Step 12 [no] ip router ospf default area 0.0.0.0 Example: apic1(config-spine-if)# ip router ospf default area 0.0.0.0 Step 13 [no] ip ospf cost cost Example: apic1(config-spine-if)# ip ospf cost 1 Step 14 Return to spine configuration mode. exit Example: apic1(config-spine-if)# exit Step 15 Repeat steps Step 6, on page 244 through Step 14, on page 245 to add any additional interfaces. Step 16 [no] router ospf default Example: apic1(config-spine)# router ospf default Step 17 [no] vrf member tenant infra vrf vrf-name Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 245 Configuring Layer 3 External Connectivity Configuring Spine Interfaces and OSPF for a Multipod Fabric Command or Action Purpose Example: apic1(config-spine-ospf)# vrf member tenant infra vrf overlay-1 Step 18 [no] area area loopback ip-address Example: Advertise the loopback address through OSPF. This address is used by BGP EVPN sessions for peering. apic1(config-spine-ospf-vrf)# area 0.0.0.0 loopback 201.201.201.201 Step 19 [no] area area interpod peering Example: apic1(config-spine-ospf-vrf)# area 0.0.0.0 interpod peering Step 20 exit Enable inter-pod peering on the OSPF area, which will set up BGP EVPN sessions automatically using the loopback address advertised by OSPF. Return to OSPF configuration mode. Example: apic1(config-spine-ospf-vrf)# exit Step 21 exit Return to spine configuration mode. Example: apic1(config-spine-ospf)# exit Step 22 exit Return to global configuration mode. Example: apic1(config-spine)# exit Step 23 Repeat steps Step 2, on page 244 through Step 22, on page 246 to configure additional spine switches. apic1# configure # CONFIGURE FIRST SPINE apic1(config)# spine 201 apic1(config-spine)# vrf context tenant infra vrf overlay-1 apic1(config-spine-vrf)# router-id 201.201.201.201 apic1(config-spine-vrf)# exit apic1(config-spine)# interface ethernet 1/1 apic1(config-spine-if)# vlan-domain member l3Dom apic1(config-spine-if)# exit apic1(config-spine)# interface ethernet 1/1.4 apic1(config-spine-if)# vrf member tenant infra vrf overlay-1 apic1(config-spine-if)# ip address 201.1.1.1/30 apic1(config-spine-if)# ip router ospf default area 0.0.0.0 apic1(config-spine-if)# ip ospf cost 1 apic1(config-spine-if)# exit apic1(config-spine)# interface ethernet 1/2 apic1(config-spine-if)# vlan-domain member l3Dom apic1(config-spine-if)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 246 Configuring Layer 3 External Connectivity Cisco APIC Quality of Service apic1(config-spine)# interface ethernet 1/2.4 apic1(config-spine-if)# vrf member tenant infra vrf overlay-1 apic1(config-spine-if)# ip address 201.2.1.1/30 apic1(config-spine-if)# ip router ospf default area 0.0.0.0 apic1(config-spine-if)# ip ospf cost 1 apic1(config-spine-if)# exit apic1(config-spine)# router ospf default apic1(config-spine-ospf)# vrf member tenant infra vrf overlay-1 apic1(config-spine-ospf-vrf)# area 0.0.0.0 loopback 201.201.201.201 apic1(config-spine-ospf-vrf)# area 0.0.0.0 interpod peering apic1(config-spine-ospf-vrf)# exit apic1(config-spine-ospf)# exit apic1(config-spine)# exit # CONFIGURE SECOND SPINE apic1(config)# spine 202 apic1(config-spine)# vrf context tenant infra vrf overlay-1 apic1(config-spine-vrf)# router-id 202.202.202.202 apic1(config-spine-vrf)# exit apic1(config-spine)# interface ethernet 1/2 apic1(config-spine-if)# vlan-domain member l3Dom apic1(config-spine-if)# exit apic1(config-spine)# interface ethernet 1/2.4 apic1(config-spine-if)# vrf member tenant infra vrf overlay-1 apic1(config-spine-if)# ip address 202.1.1.1/30 apic1(config-spine-if)# ip router ospf default area 0.0.0.0 apic1(config-spine-if)# ip ospf cost 1 apic1(config-spine-if)# exit apic1(config-spine)# router ospf default apic1(config-spine-ospf)# vrf member tenant infra vrf overlay-1 apic1(config-spine-ospf-vrf)# area 0.0.0.0 loopback 202.202.202.202 apic1(config-spine-ospf-vrf)# area 0.0.0.0 interpod peering apic1(config-spine-ospf-vrf)# exit apic1(config-spine-ospf)# exit apic1(config-spine)# exit # CONFIGURE ADDITIONAL SPINES Cisco APIC Quality of Service CoS Preservation Preserving 802.1P Class of Service Settings APIC enables preserving 802.1P class of service (CoS) settings within the fabric. Enable the fabric global QoS policy dot1p-preserve option to guarantee that the CoS value in packets which enter and transit the ACI fabric is preserved. 802.1P CoS preservation is supported in single pod and multipod topologies. In multipod topologies, CoS Preservation can be used where you want to preserve the QoS priority settings of 802.1P traffic entering POD 1 and egressing out of POD 2, but you are not concerned with preserving the CoS/DSCP settings in interpod network (IPN) traffic between the pods. To preserve CoS/DSCP settings when multipod traffic is transitting an IPN, use a DSCP policy. For more information, see Preserving QoS Priority Settings in a Multipod Fabric, on page 250. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 247 Configuring Layer 3 External Connectivity CoS Preservation Observe the following 801.1P CoS preservation guidelines and limitations: • The current release can only preserve the 802.1P value within a VLAN header. The DEI bit is not preserved. • For VXLAN encapsulated packets, the current release will not preserve the 802.1P CoS value contained in the outer header. • 802.1P is not preserved when the following configuration options are enabled: ◦Multipod QoS (using a DSCP policy) is enabled. ◦Contracts are configured that include QoS. ◦Dynamic packet prioritization is enabled. ◦The outgoing interface is on a FEX. ◦Preserving QoS CoS priority settings is not supported when traffic is flowing from an EPG with isolation enforced to an EPG without isolation enforced. ◦A DSCP QoS policy is configured on a VLAN EPG and the packet has an IP header. DSCP marking can be set at the filter level on the following with the precedence order from the innermost to the outermost: ◦Contract ◦Subject ◦In Term ◦Out Term Note When specifying vzAny for a contract, external EPG DSCP values are not honored because vzAny is a collection of all EPGs in a VRF, and EPG specific configuration cannot be applied. If EPG specific target DSCP values are required, then the external EPG should not use vzAny. Preserving QoS CoS Settings Using the NX-OS Style CLI To ensure that QoS priority settings are handled the same for traffic entering and transitting a single-pod fabric, or for traffic entering one pod and egressing another in a multipod fabric, enable CoS preservation using the commands in the following steps: Note Enabling CoS preservation applies a default mapping of the CoS priorities to DSCP levels to the various traffic types. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 248 Configuring Layer 3 External Connectivity Multipod QoS Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# Step 2 configure Enables CoS preservation. qos preserve cos Example: apic1(config)# qos preserve cos Multipod QoS Enabling Multipod QoS With a DSCP Policy, Using the NX-OS Style CLI Create a DSCP map (known as a DSCP policy in the APIC GUI) to guarantee QoS priority settings in a multipod topology. The mappings must be unique within the policy. Configure a DSCP map with custom mappings for traffic streams with the following steps: Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. apic1# Step 2 tenant tenant-name configure Enters tenant configuration mode for the infra tenant. apic1(config)# Step 3 Step 4 tenant infra qos dscp-map Configures the DSCP map. dscp-translation-policy-name apic1(config-tenant)# set dscp-code qos dscp-map default Sets the custom DSCP mappings, similar to the following example. The mappings must all be unique within a DSCP map. Note For traffic passing through the IPN, do not map any DSCP value to COS6 (except traceroute traffic). apic1(config-qos-cmap# apic1(config-qos-cmap# apic1(config-qos-cmap# apic1(config-qos-cmap# apic1(config-qos-cmap# apic1(config-qos-cmap# set set set set set set dscp-code dscp-code dscp-code dscp-code dscp-code dscp-code control CS3 span CS5 level1 CS0 level2 CS1 level3 CS2 policy CS4 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 249 Configuring Layer 3 External Connectivity Multipod QoS Command or Action Purpose apic1(config-qos-cmap# set dscp-code traceroute CS6 Step 5 no shutdown Enables the DSCP map. apic1(config-qos-cmap)# no shutdown Preserving QoS Priority Settings in a Multipod Fabric This topic describes how to guarantee QoS priority settings in a multipod topology, where devices in the interpod network are not under APIC management, and may modify 802.1P settings in traffic transitting their network. Note You can alternatively use CoS Preservation where you want to preserve the QoS priority settings of 802.1P traffic entering POD 1 and egressing out of POD 2, but you are not concerned with preserving the CoS/DSCP settings in interpod network (IPN) traffic between the pods. For more information, see Preserving 802.1P Class of Service Settings, on page 247. Figure 20: Multipod Topology As illustrated in this figure, traffic between pods in a multipod topology passes through an IPN, which may not be under APIC management. When an 802.1P frame is sent from a spine or leaf switch in POD 1, the devices in the IPN may not preserve the CoS setting in 802.1P frames. In this situation, when the frame reaches a POD 2 spine or leaf switch, it has the CoS level assigned by the IPN device, instead of the level assigned at the source in POD 1. Use a DSCP policy to ensure that the QoS priority levels are preserved in this case. Configure a DSCP policy to preserve the QoS priority settings in a multipod topology, where there is a need to do deterministic mapping from CoS to DSCP levels for different traffic types, and you want to prevent the Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 250 Configuring Layer 3 External Connectivity Translating QoS Ingress Markings to Egress Markings devices in the IPN from changing the configured levels. With a DSCP policy enabled, APIC converts the CoS level to a DSCP level, according to the mapping you configure. When a frame is sent from POD 1 (with the PCP level mapped to a DSCP level), when it reaches POD 2, the mapped DSCP level is then mapped back to the original PCP CoS level. Translating QoS Ingress Markings to Egress Markings Translating QoS Ingress Markings to Egress Markings APIC enables translating the 802.1P CoS field (Class of Service) based on the ingress DSCP value. 802.1P CoS translation is supported only if DSCP is present in the IP packet and dot1P is present in the Ethernet frames. This functionality enables the ACI Fabric to classify the traffic for devices that classify the traffic based only on the CoS value. It allows mapping the dot1P CoS value based on the ingress dot1P value. It is mainly applicable for Layer 2 packets, which do not have an IP header. Observe the following 802.1P CoS translation guidelines and limitations: • Enable the fabric global QoS policy dot1p-preserve option. • 802.1P CoS translation is not supported on external L3 interfaces. • 802.1P CoS translation is supported only if the egress frame is 802.1Q encapsulated. 802.1P CoS translation is not supported when the following configuration options are enabled: • Contracts are configured that include QoS. • The outgoing interface is on a FEX. • Multipod QoS using a DSCP policy is enabled. • Dynamic packet prioritization is enabled. • If an EPG is configured with intra-EPG endpoint isolation enforced. • If an EPG is configured with allow-microsegmentation enabled. Translating QoS CoS Settings Using the NX-OS CLI Create a custom QoS policy and then associate the policy with an EPG using the following commands: Before You Begin Create the tenant, application, and EPGs that will consume the custom QoS policy. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 251 Configuring Layer 3 External Connectivity Translating QoS Ingress Markings to Egress Markings Command or Action Purpose Note Example: apic1#configure Step 2 tenant tenant-name Enter the commands listed in steps 1-5 to create a custom QoS policy. Enters tenant configuration mode for the tenant. Example: apic1(config)#tenant t001 Step 3 policy-map type qos QoS-policy-name Creates QoS policy. Example: apic1(config-tenant)#policy-map type qos baz Step 4 match dscp AF23 AF31 set-cos 6 Sets DCSP value and target QoS value. Example: apic1(config-tenant-pmap-qos)#match dscp AF23 AF31 set-cos set-cos 6 Step 5 exit Returns to the tenant configuration mode. Example: apic1(config-tenant-pmap-qos)#exit Step 6 application app-name Creates an application profile. Enter the commands listed in steps 6-9 to associate the custom QoS policy with an EPG. Creates an EPG in the application profile. Note Example: apic1(config-tenant)#application ap2 Step 7 epg epg-name Example: apic1(config-tenant-app)# epg ep2 Step 8 service-policy policy-name Associates the EPG to the policy. Example: apic1(config-tenant-app-epg)#service-policy baz Step 9 exit Returns to the tenant configuration mode. Example: apic1(config-tenant-app-epg)#exit Step 10 external-l2epg epg-name Creates an external layer 2 EPG. Note Example: apic1(config-tenant)#external-l2 epg myout:12 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 252 Enter the commands listed in steps 10-12- to associate the custom QoS policy with an external L2 EPG. Configuring Layer 3 External Connectivity Translating QoS Ingress Markings to Egress Markings Step 11 Command or Action Purpose service-policy policy-name Associates the EPG to the policy. Example: apic1(config-tenant-12ext-epg)#service-policy baz Step 12 exit Returns to the tenant configuration mode. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 253 Configuring Layer 3 External Connectivity Translating QoS Ingress Markings to Egress Markings Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 254 CHAPTER 7 Configuring Management Interfaces • Configuring Out-of-Band Management Access, page 255 • Configuring Inband Management Access, page 257 Configuring Out-of-Band Management Access To configure out-of-band (OOB) management access for controllers, leaf switches, or spine switches, these steps must be performed: • Configure the OOB management IP address and gateway on the management interface • Allow access from the necessary external subnets • Allow the necessary protocols on the management ports Before You Begin The APIC out-of-band management connection link must be 1 Gbps. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 {controller apic-number-or-range | switch Specifies the controller or switch to be configured. You can enter a range of controllers or switches node-id[-node-id-or-range]} using dashes or commas. Example: apic1(config)# controller 1-3 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 255 Configuring Management Interfaces Configuring Out-of-Band Management Access Step 3 Command or Action Purpose interface mgmt0 The mgmt0 interface provides out-of-band management, which enables you to manage the device by its IPv4 address. Example: apic1(config-controller)# interface mgmt0 Step 4 ip address addr/mask gateway addr Example: apic1(config-controller-if)# ip address-range 172.23.48.16/21 gateway 172.23.48.1 Configures the IP address and gateway for OOB management. If you specified more than one controller or switch, the command becomes ip address-range and IP addresses are assigned sequentially beginning with the address specified in this command. Note Step 5 The APIC management interface does not support an IPv6 address and cannot connect to an external IPv6 server through this interface. exit Example: apic1(config-controller-if)# exit Step 6 exit Example: apic1(config-controller)# exit Step 7 tenant mgmt System Management policies are configured under a special tenant called mgmt. Example: apic1(config)# tenant mgmt Step 8 external-l3 epg default oob-mgmt Enters the configuration mode of the out-of-band management EPG. Example: apic1(config-tenant)# external-l3 epg default oob-mgmt Step 9 match ip addr/mask Example: Provides access control for out-of-band management interface to external management subnets. apic1(config-tenant-l3ext-epg)# match ip 192.0.20.0/24 Step 10 exit Example: apic1(config-tenant-l3ext-epg)# exit Step 11 access-list oob-default Example: apic1(config-tenant)# access-list oob-default Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 256 Configures the access list filter for the OOB default policy. Configuring Management Interfaces Configuring Inband Management Access Step 12 Command or Action Purpose match tcp dest 443 Allows access on the management interface for HTTPS traffic (TCP/443). Example: apic1(config-tenant-acl)# match tcp dest 443 Step 13 Allows access on the management interface for SSH traffic (TCP/22). match tcp dest 22 Example: apic1(config-tenant-acl)# match tcp dest 22 Examples This example shows how to configure out-of-band management access for three APIC controllers. In this example, the three controllers are assigned sequential IP addresses, with controller 1 at 172.23.48.16/21, controller 2 at 172.23.48.17/21, and controller 3 at 172.23.48.18/21. apic1# configure apic1(config)# controller 1-3 apic1(config-controller)# interface mgmt0 apic1(config-controller-if)# ip address-range 172.23.48.16/21 gateway 172.23.48.1 apic1(config-controller-if)# exit apic1(config-controller)# exit apic1(config)# tenant mgmt apic1(config-tenant)# external-l3 epg default oob-mgmt apic1(config-tenant-l3ext-epg)# match ip 192.0.20.0/24 apic1(config-tenant-l3ext-epg)# exit apic1(config-tenant)# access-list oob-default apic1(config-tenant-acl)# match tcp dest 443 apic1(config-tenant-acl)# match tcp dest 22 This example shows how to configure out-of-band management access for a leaf or spine switch. apic1# configure apic1(config)# switch 101 apic1(config-switch)# interface mgmt0 apic1(config-switch-if)# ip address 172.23.48.101/21 gateway 172.23.48.1 Configuring Inband Management Access Configuring Inband Management Access to a Switch from an Outside Network To configure inband (IB) management access for leaf switches or spine switches, these steps must be performed: • Configure the inband management IP address and gateway on the inband management interface • Create or specify a VLAN domain for external inband connectivity • Add the external management station interface to the VLAN domain Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 257 Configuring Management Interfaces Configuring Inband Management Access to a Switch from an Outside Network • Allow the necessary protocols on the management ports Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 switch switch-id-or-range Example: Specifies the switch to be configured. You can enter a range of switches using dashes or commas. apic1(config)# switch 101 Step 3 interface inband-mgmt0 The inband-mgmt0 interface provides inband management. Example: apic1(config-switch)# interface inband-mgmt0 Step 4 ip address addr/mask gateway addr Example: apic1(config-switch-if)# ip address 10.13.1.1/24 gateway 10.13.1.254 Step 5 Configures the IP address and gateway for inband management. If you specified more than one switch, the command becomes ip address-range and IP addresses are assigned sequentially beginning with the address specified in this command. exit Example: apic1(config-switch-if)# exit Step 6 exit Example: apic1(config-switch)# exit Examples This example shows how to configure inband management for a switch from a management station on an external network.. apic1# configure apic1(config)# switch 101 apic1(config-switch)# interface inband-mgmt0 apic1(config-switch-if)# ip address 10.13.1.1/24 gateway 10.13.1.254 apic1(config-switch-if)# exit apic1(config-switch)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 258 Configuring Management Interfaces Configuring Inband Management Access to a Controller from an Outside Network What to Do Next • Configure inband (IB) management connectivity to the management station. • Allow the necessary protocols (HTTPS and SSH) on the inbound management port. Configuring Inband Management Access to a Controller from an Outside Network To configure inband (IB) management access for controllers, these steps must be performed: • Configure the inband management IP address and gateway on the inband management interface • Create a VLAN domain for external inband connectivity • Allow the VLAN on the port connected to the controller Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 controller controller-id-or-range Example: Specifies the controller to be configured. You can enter a range of controllers using dashes or commas. apic1(config)# controller 1-3 Step 3 interface inband-mgmt0 The inband-mgmt0 interface provides inband management. Example: apic1(config-controller)# interface inband-mgmt0 Step 4 ip address addr/mask gateway addr Example: apic1(config-controller-if)# ip address-range 10.13.1.1/24 gateway 10.13.1.254 Step 5 vlan vlan-id Example: apic1(config-controller-if)# vlan 10 Step 6 Configures the IP address and gateway for inband management. If you specified more than one controller or switch, the command becomes ip address-range and IP addresses are assigned sequentially beginning with the address specified in this command. Assigns a controller VLAN which is enabled on the port connected to the controller. For multiple controllers, all controllers must use the same VLAN. exit Example: apic1(config-controller-if)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 259 Configuring Management Interfaces Configuring Inband Management Access to a Controller from an Outside Network Command or Action Step 7 Purpose exit Example: apic1(config-controller)# exit Step 8 vlan-domain domain-name Creates and enters the configuration mode for the VLAN domain. Example: apic1(config)# vlan-domain apic-inband Step 9 vlan vlan-id Assigns the controller VLAN to the VLAN domain. Example: apic1(config-vlan)# vlan 10 Step 10 exit Returns to global configuration mode. Example: apic1(config-vlan)# exit Step 11 leaf node-id Specifies the leaf switch to which the controller connected. Example: apic1(config)# leaf 102 Step 12 interface slot/port Specifies the port to which the controller is connected. Example: apic1(config-leaf)# interface eth 1/1 Step 13 vlan-domain member apic-inband Example: apic1(config-leaf-if)# vlan-domain member apic-inband Step 14 exit Example: apic1(config-leaf-if)# exit Step 15 exit Example: apic1(config-leaf)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 260 Configures controller connectivity to inband management. Configuring Management Interfaces Configuring Inband Management Connectivity to the Management Station Examples This example shows how to configure inband management for a controller from a management station on an external network. APIC controller 1 is connected to port Ethernet 1/1 on Leaf 101, and VLAN 10 is used for the controller's inband connectivity. apic1# configure apic1(config)# controller 1-3 apic1(config-controller)# interface inband-mgmt0 apic1(config-controller-if)# ip address-range 10.13.1.1/24 gateway 10.13.1.254 apic1(config-controller-if)# vlan 10 apic1(config-controller-if)# exit apic1(config-controller)# exit # CREATE A VLAN DOMAIN FOR THE APIC INBAND VLAN apic1(config)# vlan-domain apic-inband apic1(config-vlan)# vlan 10 apic1(config-vlan)# exit # ALLOW THE VLAN ON THE PORT CONNECTED TO THE CONTROLLER apic1(config)# leaf 101 apic1(config-leaf)# interface eth 1/1 apic1(config-leaf-if)# vlan-domain member apic-inband apic1(config-leaf-if)# exit apic1(config-leaf)# exit What to Do Next • Configure inband (IB) management connectivity to the management station. • Allow the necessary protocols (HTTPS and SSH) on the inbound management port. Configuring Inband Management Connectivity to the Management Station To configure inband (IB) management connectivity to the management station, these steps must be performed: • Create or specify a VLAN domain for external inband connectivity • Add the external management station interface to the VLAN domain Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 vlan-domain domain-name Creates and enters the configuration mode for the VLAN domain. Example: apic1(config)# vlan-domain external-inband Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 261 Configuring Management Interfaces Configuring Inband Management Connectivity to the Management Station Step 3 Command or Action Purpose vlan vlan-id Assigns a VLAN to the domain. Example: apic1(config-vlan)# vlan 11 Step 4 exit Returns to global configuration mode. Example: apic1(config-vlan)# exit Step 5 leaf node-id Specifies the leaf switch to which the management station is connected. Example: apic1(config)# leaf 102 Step 6 interface slot/port Specifies the port to which the management station is connected. Example: apic1(config-leaf)# interface eth 1/2 Step 7 vlan-domain member external-inband Configures external layer2 connectivity to inband management. Example: apic1(config-leaf-if)# vlan-domain member external-inband Step 8 switchport trunk allowed vlan vlan-id inband-mgmt gateway-ip/mask Example: apic1(config-leaf-if)# switchport trunk allowed vlan 11 inband-mgmt 179.10.1.254/24 Step 9 Configures external layer2 connectivity to inband management. The specified IP address is the gateway address used by the external management station and the gateway functionality is provided by the ACI fabric. exit Example: apic1(config-leaf-if)# exit Step 10 exit Example: apic1(config-leaf)# exit Examples This example shows how to configure inband management connectivity to the management station. # CREATE A VLAN DOMAIN FOR EXTERNAL CONNECTIVITY TO INBAND MANAGEMENT apic1# configure apic1(config)# vlan-domain external-inband apic1(config-vlan)# vlan 11 apic1(config-vlan)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 262 Configuring Management Interfaces Configuring Inband Management Contract to Open HTTPS/SSH Ports # CONFIGURE LAYER 2 CONNECTIVITY FROM THE MANAGEMENT STATION INTERFACE TO INBAND MANAGEMENT apic1(config)# leaf 102 apic1(config-leaf)# interface eth 1/2 apic1(config-leaf-if)# vlan-domain member external-inband apic1(config-leaf-if)# switchport trunk allowed vlan 11 inband-mgmt 179.10.1.254/24 apic1(config-leaf-if)# exit apic1(config-leaf)# exit What to Do Next • Allow the necessary protocols (HTTPS and SSH) on the inbound management port. Configuring Inband Management Contract to Open HTTPS/SSH Ports Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 System Management policies are configured under a special tenant called mgmt. tenant mgmt Example: apic1(config)# tenant mgmt Step 3 access-list inband-default Configures the access list filter for the inband default policy. Example: apic1(config-tenant)# access-list inband-default Step 4 Allows access on the management interface for HTTPS traffic (TCP/443). match tcp dest 443 Example: apic1(config-tenant-acl)# match tcp dest 443 Step 5 Allows access on the management interface for SSH traffic (TCP/22). match tcp dest 22 Example: apic1(config-tenant-acl)# match tcp dest 22 Examples This example shows how to allow HTTPS and SSH access to the inband management port. apic1# configure apic1(config)# tenant mgmt apic1(config-tenant)# access-list inband-default Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 263 Configuring Management Interfaces Configuring Inband Management Contract to Open HTTPS/SSH Ports apic1(config-tenant-acl)# match tcp dest 443 apic1(config-tenant-acl)# match tcp dest 22 apic1(config-tenant-acl)# exit apic1(config-tenant)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 264 CHAPTER 8 Configuring Security • About Security Configuration, page 265 • Configuring AAA, page 266 • Configuring Security Servers, page 269 • Configuring the Password Policy, page 276 • Configuring Users, page 279 • Configuring Public Key Infrastructure, page 283 • Configuring Communication Policies, page 288 • Configuring AES Encryption, page 293 • Configuring Fabric Secure Mode, page 294 • Configuring COOP Authentication, page 295 • Configuring FIPS, page 296 • Configuring Control Plane Policing, page 298 • Configuring First Hop Security, page 301 About Security Configuration Access control is the way you control who is allowed access to the network server and what services they are allowed to use once they have access. Authentication, authorization, and accounting (AAA) network security services provide the primary framework through which you set up access control on APIC. Overview of the AAA Configuration To configure security on APIC using AAA, follow this process: 1 To use a separate security server, configure security protocol parameters using the radius-server, ldap-server, or tacacs-server configuration commands. 2 Define the method lists for authentication by using an aaa authentication command. 3 Apply the method lists to a particular interface or line, if required. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 265 Configuring Security Configuring AAA 4 (Optional) Configure authorization using the aaa authentication command. Login Authentication Using a Local Password Use the aaa authentication login command with the method argument to specify that APIC will use the local username database for authentication. For example, to specify the local username database as the method of user authentication at login when no other method list has been defined, enter the following commands: apic1# configure apic1(config)# aaa authentication login default apic1(config-default)# realm local For information about adding users into the local username database, refer to the section “Configuring a Locally Authenticated User.” Login Authentication Using a Remote Server Use the aaa authentication login command with the server radius/tacacs/ldap method to specify RADIUS/TACACS+/LDAP as the login authentication method. For example, to specify RADIUS as the method of user authentication at login when no other method list has been defined, enter the following commands: apic1# configure apic1(config)# aaa authentication login default apic1(config-default)# realm radius Before you can use RADIUS as the login authentication method, you need to enable communication with the RADIUS security server, same is true for TACACS+ or LDAP. For more information about establishing communication with a remote security server, see the appropriate chapter: • "Configuring a RADIUS Server" • "Configuring a TACACS+ Server" • "Configuring an LDAP Server" Configuring AAA Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 aaa authentication login console Example: apic1(config)# aaa authentication login console Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 266 Enters console configuration mode for users accessing APIC through the console. Configuring Security Configuring AAA Step 3 Command or Action Purpose [no] realm {ldap | local | radius | tacacs} Specifies the authentication method. Example: apic1(config-console)# realm radius Step 4 [no] group group-name Specifies an authentication server group. Example: apic1(config-console)# group radiusGroup5 Step 5 Returns to global configuration mode. exit Example: apic1(config-console)# exit Step 6 aaa authentication login default Enters the configuration mode for default login authentication. Example: apic1(config)# aaa authentication login default Step 7 [no] realm {ldap | local | radius | tacacs} Specifies the authentication method. Example: apic1(config-default)# realm radius Step 8 [no] group group-name Specifies an authentication server group. Example: apic1(config-default)# group radiusGroup Step 9 Returns to global configuration mode. exit Example: apic1(config-default)# exit Step 10 aaa authentication login domain {domain-name | fallback} Enters the configuration mode for default login authentication. A login domain specifies the authentication domain for a user. Example: apic1(config)# aaa authentication login domain cisco Step 11 [no] realm {ldap | local | none | radius | tacacs} Specifies the authentication method. Example: apic1(config-domain)# realm radius Step 12 [no] group group-name Specifies an authentication server group. Example: apic1(config-domain)# group radiusGroup Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 267 Configuring Security Configuring AAA Step 13 Command or Action Purpose exit Returns to global configuration mode. Example: apic1(config-domain)# exit Step 14 aaa banner text Example: Specifies the informational banner to be displayed before the user login. The banner must be contained in single quotes. apic1(config)# aaa banner 'Welcome to APIC' Step 15 aaa group {ldap | radius | tacacs} group-name Creates or configures an authentication server group. Example: apic1(config)# aaa group radius radiusGroup Step 16 [no] server {ip-address | hostname} priority Adds a server to the authentication server group and specifies its priority within the priority-number server group. The priority can be between 0 and 17. Example: apic1(config-radius)# server 192.0.20.71 priority 2 Step 17 exit Returns to global configuration mode. Example: apic1(config-radius)# exit Step 18 aaa scvmm-certificate certificate-name Specifies an SCVMM certificate. See the Cisco ACI Virtualization Guide. Example: apic1(config)# aaa scvmm-certificate myScvmmCert Step 19 aaa user default-role {assign-default-role | no-login} Example: apic1(config)# aaa user default-role assign-default-role Specifies how to respond when remote users who do not have a user role attempt to log in to APIC. The action can be either of these options: • assign-default-role—Remote users who do not have a user role are assigned a default role. • no-login—Remote users who do not have a user role cannot log in. Step 20 show aaa authentication Example: apic1(config)# show aaa authentication Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 268 Displays configured AAA methods. Configuring Security Configuring Security Servers Step 21 Command or Action Purpose show aaa groups Displays configured AAA server groups. Example: apic1(config)# show aaa groups Examples This example shows how to configure AAA. apic1# configure terminal apic1(config)# aaa authentication login console apic1(config-console)# realm local apic1(config-console)# exit apic1(config)# aaa authentication login default apic1(config-default)# realm radius apic1(config-default)# group radiusGroup5 apic1(config-default)# exit apic1(config)# aaa authentication login domain cisco apic1(config-domain)# realm none apic1(config-domain)# exit apic1(config)# aaa banner 'Welcome to APIC' apic1(config)# aaa group radius radiusGroup apic1(config-radius)# server 192.0.20.71 priority 2 apic1(config-radius)# exit apic1(config)# aaa user default-role assign-default-role apic1(config)# show aaa authentication Default : radius Console : local apic1(config)# show aaa groups Total number of Groups : 1 RadiusGroups : radiusGroup5 TacacsGroups : LdapGroups : Configuring Security Servers Configuring a RADIUS Server Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 269 Configuring Security Configuring a RADIUS Server Step 2 Command or Action Purpose [no] radius-server retries count Specifies how many times APIC transmits each RADIUS request to the server before giving up. The range is 0 to 5. Example: Step 3 apic1(config)# radius-server retries 1 In the global configuration mode, this command applies to all RADIUS servers unless overridden in the specific RADIUS host configuration. [no] radius-server timeout seconds Specifies the number of seconds APIC waits for a reply to a RADIUS request before retransmitting the request. Example: In the global configuration mode, this command applies to all RADIUS servers unless overridden in the specific RADIUS host configuration. apic1(config)# radius-server timeout 5 Step 4 [no] radius-server host {ip-address | Specifies the IP address or hostname of the RADIUS server. hostname} Example: apic1(config)# radius-server host 192.0.20.71 Step 5 [no] retries count Example: apic1(config-host)# retries 2 (Optional) For this RADIUS server, specifies how many times APIC transmits each RADIUS request to the server before giving up. The range is 0 to 5. If no retry count is set, the global value is used. Step 6 [no] timeout seconds Example: apic1(config-host)# timeout 3 (Optional) For this RADIUS server, specifies the number of seconds APIC waits for a reply to a RADIUS request before retransmitting the request. If no timeout is set, the global value is used. Step 7 [no] descr text Example: apic1(config-host)# descr "My primary RADIUS server" Step 8 [no] key key-value Example: (Optional) Provides descriptive information about this RADIUS server. The text can be up to 128 alphanumeric characters. If the text contains spaces, it must be enclosed by single or double quotes. Specifies the shared secret text string used between APIC and this RADIUS server for authentication. The key can be up to 32 characters. apic1(config-host)# key myRaDiUSpassWoRd Step 9 [no] port port-number Example: apic1(config-host)# port 1812 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 270 Specifies a UDP port on this RADIUS server to be used solely for authentication. Configuring Security Configuring a RADIUS Server Command or Action Step 10 Purpose [no] protocol {chap | mschap | pap} Specifies the RADIUS server protocol for authentication. Example: apic1(config-host)# protocol pap Step 11 Returns to global configuration mode. exit Example: apic1(config-host)# Step 12 (Optional) Displays the RADIUS server information. show radius-server Example: apic1(config)# show radius-server Examples This example shows how to configure RADIUS settings globally and on one RADIUS server. apic1# configure apic1(config)# radius-server retries 1 apic1(config)# radius-server timeout 5 apic1(config)# radius-server host 192.0.20.71 apic1(config-host)# retries 2 apic1(config-host)# timeout 3 apic1(config-host)# descr "My primary RADIUS server" apic1(config-host)# key myRaDiUSpassWoRd apic1(config-host)# port 1812 apic1(config-host)# protocol pap apic1(config-host)# exit apic1(config)# show radius-server timeout : 5 retries : 1 Total number of servers : 1 Hostname Port Protocol Timeout Retries User Descr : : : : : : : 192.0.20.71 1812 pap 3 2 test My primary RADIUS server Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 271 Configuring Security Configuring a TACACS+ Server Configuring a TACACS+ Server Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] tacacs-server retries count Example: Step 3 Step 4 Specifies how many times APIC transmits each TACACS+ request to the server before giving up. The range is 0 to 5. apic1(config)# tacacs-server retries 1 In the global configuration mode, this command applies to all TACACS+ servers unless overridden in the specific TACACS+ host configuration. [no] tacacs-server timeout seconds Specifies the number of seconds APIC waits for a reply to a TACACS+ request before retransmitting the request. Example: apic1(config)# tacacs-server timeout 5 In the global configuration mode, this command applies to all TACACS+ servers unless overridden in the specific TACACS+ host configuration. [no] tacacs-server host {ip-address | hostname} Specifies the IP address or hostname of the TACACS+ server. Example: apic1(config)# tacacs-server host 192.0.20.71 Step 5 [no] retries count Example: apic1(config-host)# retries 2 (Optional) For this TACACS+ server, specifies how many times APIC transmits each TACACS+ request to the server before giving up. The range is 0 to 5. If no retry count is set, the global value is used. Step 6 [no] key Example: apic1(config-host)# key Enter key: myTacAcSpassWoRd Enter key again: myTacAcSpassWoRd Step 7 [no] port port-number Example: apic1(config-host)# port 49 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 272 Specifies the shared secret text string used between APIC and this TACACS+ server for authentication. The key can be up to 32 characters. For increased security, entering the key value is interactive. Specifies a UDP port on this TACACS+ server to be used for TACACS+ accounting messages. Configuring Security Configuring an LDAP Server Step 8 Command or Action Purpose [no] protocol {chap | mschap | pap} Specifies the TACACS+ server protocol for authentication. Example: apic1(config-host)# protocol pap Step 9 Returns to global configuration mode. exit Example: apic1(config-host)# Step 10 (Optional) Displays the TACACS+ server information. show tacacs-server Example: apic1(config)# show tacacs-server Examples This example shows how to configure TACACS+ settings globally and on one TACACS+ server. apic1# configure apic1(config)# tacacs-server retries 1 apic1(config)# tacacs-server timeout 5 apic1(config)# tacacs-server host 192.0.20.72 apic1(config-host)# retries 2 apic1(config-host)# timeout 3 apic1(config-host)# key myTaCaCspassWoRd apic1(config-host)# port 49 apic1(config-host)# protocol pap apic1(config-host)# exit apic1(config)# show tacacs-server timeout : 5 retries : 1 Total number of servers : 1 Hostname Port Protocol Timeout Retries User : : : : : : 192.0.20.72 1812 pap 3 2 test Configuring an LDAP Server Some ldap-server commands can be entered in either the global configuration mode or in the configuration mode for a specific LDAP host. In the global configuration mode, the command applies to all LDAP servers unless overridden in the specific LDAP host configuration. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 273 Configuring Security Configuring an LDAP Server Procedure Step 1 Command or Action Purpose configure terminal Enters global configuration mode. Example: switch# configure terminal Step 2 [no] ldap-server host {ip-address | hostname} Example: Specifies the IP address or hostname of the LDAP server and enters the configuration mode of that server. apic1(config)# ldap-server host 192.0.20.73 Step 3 [no] ldap-server attribute attribute-name Specifies an LDAP endpoint attribute to be used as the CiscoAVPair. Example: In the global configuration mode, this command applies to all LDAP servers unless overridden in the specific LDAP host configuration. apic1(config-host)# ldap-server attribute memberOf Step 4 [no] ldap-server basedn Example: apic1(config-host)# ldap-server basedn DC=sampledesign,DC=com Specifies the location in the LDAP hierarchy where the server should begin searching when it receives an authorization request. This can be a string of up to 127 characters. Spaces are not permitted in the string, but other special characters are allowed. In the global configuration mode, this command applies to all LDAP servers unless overridden in the specific LDAP host configuration. Step 5 [no] ldap-server binddn Example: apic1(config-host)# ldap-server binddn CN=ucsbind,OU=CiscoUsers,DC=sampledesign,DC=com Step 6 [no] ldap-server retries count Example: apic1(config-host)# ldap-server retries 1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 274 Specifies the distinguished name (DN) for an LDAP database account that has read and search permissions for all objects under the base DN. This can be a string of up to 127 characters. Spaces are not permitted in the string, but other special characters are allowed. Specifies how many times APIC transmits each LDAP request to the server before giving up. The range is 0 to 5. In the global configuration mode, this command applies to all LDAP servers unless overridden in the specific LDAP host configuration. Configuring Security Configuring an LDAP Server Step 7 Command or Action Purpose [no] ldap-server timeout seconds Specifies the number of seconds APIC waits for a reply to a LDAP request before retransmitting the request. Example: Step 8 apic1(config-host)# ldap-server timeout 30 In the global configuration mode, this command applies to all LDAP servers unless overridden in the specific LDAP host configuration. [no] ldap-server filter filter-expression Specifies a filter to filter the results of LDAP searches. The filter can contain a maximum of 63 characters. Example: Step 9 apic1(config-host)# ldap-server filter sAMAccountName=$userid In the global configuration mode, this command applies to all LDAP servers unless overridden in the specific LDAP host configuration. [no] key key-value Specifies the shared secret text string used between APIC and this LDAP server for authentication. The key can be up to 32 characters. Example: apic1(config-host)# key Enter key: myLdAppassWoRd Enter key again: myLdAppassWoRd Step 10 [no] port port-number Specifies the LDAP server port for authentication. Example: apic1(config-host)# port 389 Step 11 [no] retries count Example: apic1(config-host)# retries 2 (Optional) For this LDAP server, specifies how many times APIC transmits each LDAP request to the server before giving up. The range is 0 to 5. If no retry count is set, the global value is used. Step 12 [no] enable-ssl Enables an SSL connection with the LDAP provider. Example: apic1(config-host)# enable-ssl Step 13 [no] ssl-validation-level [permissive | strict] Sets the LDAP Server SSL Certificate validation level. Example: apic1(config-host)# ssl-validation-level permissive Step 14 [no] timeout seconds Example: apic1(config-host)# timeout 3 (Optional) For this LDAP server, specifies the number of seconds APIC waits for a reply to a LDAP request before retransmitting the request. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 275 Configuring Security Configuring the Password Policy Command or Action Purpose If no timeout is set, the global value is used. Step 15 exit Returns to global configuration mode. Example: apic1(config-host)# exit Step 16 show ldap-server Example: apic1(config)# show ldap-server Examples This example shows how to configure LDAP server settings globally and on one LDAP server. apic1# configure apic1(config)# ldap-server retries 1 apic1(config)# ldap-server timeout 30 apic1(config)# ldap-server host 192.0.20.73 apic1(config-host)# retries 2 apic1(config-host)# timeout 3 apic1(config-host)# filter sAMAccountName=$userid apic1(config-host)# key myLdAppassWoRd apic1(config-host)# ssl-validation-level permissive apic1(config-host)# enable-ssl apic1(config-host)# port 389 apic1(config-host)# exit apic1(config)# show ldap-server timeout : 30 retries : 1 filter : sAMAccountName=$userid Total number of servers : 1 Hostname Port Timeout Retries SSL SSL Level User : : : : : : : 192.0.20.73 389 3 2 yes permissive test Configuring the Password Policy The password policy configuration in this topic set the password history and password change interval properties for all locally authenticated APIC users. You cannot specify different password policies for each locally authenticated user. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 276 Configuring Security Configuring the Password Policy Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] password change-count count Sets the number of password changes allowed within the change interval. The range is 0 to 10 changes. Example: apic1(config)# password change-count 5 Step 3 [no] password change-during-interval {enable | disable} Enables or disables restricting the number of password changes a locally authenticated user can make within the change interval. Example: apic1(config)# password change-during-interval enable Step 4 [no] password change-interval hours Example: When the change-during-interval is enabled, restricts the number of password changes a locally authenticated user can make within a given number of hours. The range is 1 to 745 hours. apic1(config)# password change-interval 300 Step 5 [no] password no-change-interval Sets a minimum period before which a user cannot change the password again. The range is 1 to 745 hours. hours Example: apic1(config)# password no-change-interval 60 Step 6 password expiration-warn-time Sets a warning period before password expiration to display warning. The range is 0 to 30 days. Example: apic1(config)# password expiration-warn-time 5 Step 7 [no] password history-count count The password history count allows you to prevent locally authenticated users from reusing the same password over and over again. When this property is configured, APIC Example: stores passwords that were previously used by locally apic1(config)# password history-count 10 authenticated users up to a maximum of 15 passwords. The passwords are stored in reverse chronological order with the most recent password first to ensure that the only the oldest password can be reused when the history count threshold is reached. A user must create and use the number of passwords configured in the password history count before being able to reuse one. For example, if you set the password history Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 277 Configuring Security Configuring the Password Policy Command or Action Purpose count to 8, a locally authenticated user cannot reuse the first password until after the ninth password has expired. By default, the password history is set to 0. This value disables the history count and allows users to reuse previous passwords at any time. If necessary, you can clear a user's password history using the clear-pwd-history command in the username configuration mode for that user. Step 8 [no] password pwd-strength-check Enforces strong passwords for all users. Example: apic1(config)# password pwd-strength-check Examples This example shows how to configure global password settings for locally authenticated users. apic1# configure apic1(config)# password apic1(config)# password apic1(config)# password apic1(config)# password apic1(config)# password apic1(config)# password apic1(config)# password change-count 5 change-during-interval enable change-interval 300 no-change-interval 60 expiration-warn-time 5 history-count 10 pwd-strength-check This example shows how to prevent the password from being changed within 48 hours after a locally authenticated user changes his or her password. apic1# configure apic1(config)# password change-during-interval disable apic1(config)# password no-change-interval 48 This example shows how to allow the password to be changed a maximum of once within 24 hours after a locally authenticated user changes his or her password apic1# configure apic1(config)# password change-count 1 apic1(config)# password change-during-interval enable apic1(config)# password change-interval 24 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 278 Configuring Security Configuring Users Configuring Users Configuring a Locally Authenticated User Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 username {name | admin} Example: Creates a locally-authenticated user account or configures an existing user. The name can be a maximum of 28 characters. apic1(config)# username user5 Step 3 [no] first-name first Sets the first name of this user. Example: apic1(config-username)# first-name George Step 4 [no] last-name last Sets the last name of this user. Example: apic1(config-username)# last-name Washington Step 5 [no] email email-address Sets the email address of this user. Example: apic1(config-username)# email [email protected] Step 6 [no] phone phone-number Sets the phone number of this user. Example: apic1(config-username)# phone 14085551212 Step 7 [no] account-status {active | inactive | status} Activates or deactivates this user account. Example: apic1(config-username)# account-status active Step 8 Clears the user's password history list and allows this user to reuse previous passwords. clear-pwd-history Example: apic1(config-username)# clear-pwd-history Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 279 Configuring Security Configuring a Locally Authenticated User Step 9 Command or Action Purpose [no] expires Enables expiration of this user account at the date and time configured by the expiration command. Example: apic1(config-username)# expires Step 10 expiration date-time apic1(config-username)# expiration 2017-12-31T23:59+08:00 Sets an expiration date and time for this user account. The format is UTC Date format (YYYY-MM-DDThh:mmTZD). You must also enable expiration by configuring the expires command. password password Sets the user password. Example: Step 11 Note Example: apic1(config-username)# password c1\$c0123 Step 12 [no] pwd-lifetime days Special characters such as '$' or '!' should be escaped with a backslash ('\$') in this command to avoid misinterpretation by Bash. The escape backslash is necessary only when setting the password in this command; the user does not enter the backslash when logging in. Sets the lifetime of the user password. The range is 0 to 3650 days. Example: apic1(config-username)# pwd-lifetime 90 Step 13 [no] domain {all | common | mgmt | domain-name} Specifies or creates the AAA domain to which this user belongs. Example: apic1(config-username)# domain mySecDomain Step 14 [no] role role Creates the AAA domain role to set privilege bitmask of a user domain. Example: apic1(config-domain)# role tenant-admin Step 15 [no] priv-type {readPriv | writePriv} Creates the AAA domain role to set privilege bitmask of a user domain. Example: apic1(config-role)# priv-type writePriv Step 16 exit Example: apic1(config-role)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 280 Returns to domain configuration mode. Configuring Security Configuring a Certificate and SSH-Key for a Local User Step 17 Command or Action Purpose exit Returns to username configuration mode. Example: apic1(config-domain)# exit Step 18 show username name Displays configuration details about this user. Example: apic1(config-username)# show username user5 Examples This example shows how to configure a local user. apic1# configure terminal apic1(config)# username user5 apic1(config-username)# first-name George apic1(config-username)# last-name Washington apic1(config-username)# email [email protected] apic1(config-username)# phone 14085551212 apic1(config-username)# account-status active apic1(config-username)# domain mySecDomain apic1(config-username)# clear-pwd-history apic1(config-username)# expires apic1(config-username)# expiration 2017-12-31T23:59+08:00 apic1(config-username)# password c1$c0123 apic1(config-username)# pwd-lifetime 90 apic1(config-username)# domain mySecDomain apic1(config-domain)# role tenant-admin apic1(config-role)# priv-type writePriv apic1(config-role)# exit apic1(config-domain)# exit apic1(config-username)# show username user5 UserName : user5 First-Name : George Last-Name : Washington Email : [email protected] Acount Status : active Password strength check : yes What to Do Next To configure an SSH key or certificate for the local user, see "Configuring Certificates and SSH-Keys." Configuring a Certificate and SSH-Key for a Local User This topic describes how to configure a certificate or an SSH key so that a local user can log in without being prompted for a password. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 281 Configuring Security Configuring a Certificate and SSH-Key for a Local User Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 username {name | admin} Example: Creates a locally-authenticated user account or configures an existing user. The name can be a maximum of 28 characters. apic1(config)# username user5 Step 3 [no] certificate certificate-name Enters certificate configuration mode. Example: apic1(config-username)# certificate myCertificate Step 4 data certificate-data Sets PEM-encoded certificate. Example: apic1(config-certificate)# data -----BEGIN CERTIFICATE-----MIIC4j..... Step 5 exit Returns to username configuration mode. Example: apic1(config-certificate)# exit Step 6 [no] ssh-key ssh-key-name Sets an SSH key to log in using the SSH client without being prompted for a password. Example: apic1(config-username)# ssh-key mySSHkey Step 7 data key-data Sets the SSH key. The key can be up to 64 characters. Example: apic1(config-ssh-key)# data AAAAB3NzaC1yc2EAA...... Step 8 exit Returns to username configuration mode. Example: apic1(config-ssh-key)# exit Examples This example shows how to configure an SSH key and a certificate for a local user. apic1# configure terminal apic1(config)# username user5 apic1(config-username)# certificate myCertificate apic1(config-certificate)# data -----BEGIN CERTIFICATE-----MIIC4j..... Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 282 Configuring Security Configuring Public Key Infrastructure apic1(config-certificate)# exit apic1(config-username)# ssh-key mySSHkey apic1(config-ssh-key)# data AAAAB3NzaC1yc2EAA... apic1(config-ssh-key)# exit Configuring Public Key Infrastructure Configuring a Certificate Authority and Chain of Trust Certificate authorities (CAs) manage certificate requests and issue certificates to participating entities such as hosts, network devices, or users. APIC locally stores the self-signed root certificate of the trusted CA (or certificate chain for a subordinate CA). The stored information about a trusted CA is called the trustpoint and the CA itself is called a trustpoint CA. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] crypto ca trustpoint-name Enters configuration mode for the specified trustpoint certificate authority (CA). Example: apic1(config)# crypto ca myCA Step 3 [no] cert-chain pem-data Example: Stores the certificate chain in PEM format. Enter the entire chain of trust from the trustpoint to a trusted root authority. apic1(config-ca)# cert-chain -----BEGIN CERTIFICATE----- MIIC4jCCAoygAw..... Examples This example shows how to configure a CA. apic1# configure apic1(config)# crypto ca myCA apic1(config-ca)# cert-chain -----BEGIN CERTIFICATE----- MIIC4jCCAoygAw..... Configuring Keys and a Keyring You can obtain an identity certificate for APIC by generating an RSA key pair and associating the key pair with a trustpoint CA where APIC intends to enroll. The RSA keys are stored by APIC in a crypto keyring. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 283 Configuring Security Configuring Keys and a Keyring The APIC software allows you to generate an RSA key pair with a configurable key size (or modulus). The default key size is 512. You can also configure an RSA key-pair label. The default key label is the device fully qualified domain name (FQDN). Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] crypto keyring {default | keyring-name} Creates or configures a keyring to hold an SSL certificate. Example: apic1(config)# crypto keyring myKeyring Step 3 regen Forces regeneration of the RSA key pair. Example: apic1(config-keyring)# regen Step 4 [no] cert certificate-data Example: Imports a certificate containing a public key and signed information. The certificate data must be enclosed in quotes. apic1(config-keyring)# cert "-----BEGIN CERTIFICATE----- MIIC4jCCAoygAw..... Step 5 [no] tp certificate-name Sets a third-party certificate from a trusted source for device identity. Example: apic1(config-keyring)# tp myCertificate Step 6 [no] key key-data Creates the private key of the certificate. Example: apic1(config-keyring)# key XXXXXXXXXXXXXXXXXXXXXXX Step 7 [no] modulus {mod512 | mod1024 | mod1536 Sets the length of the encryption keys. | mod2048} Example: apic1(config-keyring)# modulus mod1024 Step 8 exit Example: apic1(config-keyring)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 284 Returns to global configuration mode. Configuring Security Generating a Certificate Signing Request Examples This example shows how to configure a keyring. apic1# configure apic1(config)# crypto keyring myKeyring apic1(config-keyring)# cert "-----BEGIN CERTIFICATE----- MIIC4jCCAoygAw..... apic1(config-keyring)# tp myCertificate apic1(config-keyring)# key XXXXXXXXXXXXXXXXXXXXXXX apic1(config-keyring)# modulus mod1024 apic1(config-keyring)# exit Generating a Certificate Signing Request A certificate signing request (CSR) is a message that an applicant sends to a CA in order to apply for a digital identity certificate. Before a CSR is created, the applicant first generates a key pair, which keeps the private key secret. The CSR contains information that identifies the applicant, such as the public key generated by the applicant. The corresponding private key is not included in the CSR, but is used to digitally sign the entire request. Before You Begin Before generating a certificate signing request (CSR), you must configure a trustpoint certificate authority (CA) and generate a key pair. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] crypto keyring {default | keyring-name} Creates or configures a keyring to hold an SSL certificate. Example: apic1(config)# crypto keyring default Step 3 Creates a certificate signing request for this keyring. csr Example: apic1(config-keyring)# csr Step 4 subj-name name Sets the fully qualified domain name or distinguished name of the requesting device. The name can be up to 64 characters. Example: apic1(config-csr)# subj-name www.exampleCorp.com Step 5 [no] cert certificate-data Example: Imports a certificate containing a public key and signed information. The certificate data must be enclosed in quotes. apic1(config-csr)# cert "-----BEGIN CERTIFICATE----- MIIC4jCCAoygAw..... Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 285 Configuring Security Generating a Certificate Signing Request Step 6 Command or Action Purpose password Sets the new password. Example: apic1(config-csr)# password Enter password: c1$c0123 Enter password again: c1$c0123 Step 7 org-name Sets the full legal name of the organization. Example: apic1(config-csr)# org-name ExampleCorp Step 8 org-unit-name Sets the department or unit name within the organization. Example: apic1(config-csr)# org-unit-name Sales Step 9 email Sets the email address of the organization contact person. Example: apic1(config-csr)# email [email protected] Step 10 locality city-name Sets the city or town of the organization. Example: apic1(config-csr)# locality SanJose Step 11 state state Sets the state or province in which the organization is located. Example: apic1(config-csr)# state CA Step 12 country country-code Sets the two-letter ISO code for the country where the organization is located. Example: apic1(config-csr)# country US Step 13 exit Returns to keyring configuration mode. Example: apic1(config-csr)# exit Examples This example shows how to generate a certificate signing request (CSR). apic1# configure apic1(config)# crypto keyring default apic1(config-keyring)# csr apic1(config-csr)# subj-name www.exampleCorp.com apic1(config-csr)# cert "-----BEGIN CERTIFICATE----- MIIC4jCCAoygAw..... apic1(config-csr)# pwd c1$c0123 apic1(config-csr)# org-name ExampleCorp Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 286 Configuring Security Configuring Webtokens apic1(config-csr)# apic1(config-csr)# apic1(config-csr)# apic1(config-csr)# apic1(config-csr)# apic1(config-csr)# org-unit-name Sales email [email protected] locality SanJose state CA country US exit What to Do Next Submit the CSR to a CA. Configuring Webtokens Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] crypto webtoken Example: apic1(config)# crypto webtoken Step 3 [no] max-validity-period hours Sets the maximum validity period for a webtoken. The range is 4 to 24 hours. Example: apic1(config-webtoken)# max-validity-period 10 Step 4 [no] session-record-flags csv-list Example: apic1(config-webtoken)# session-record-flags login,refresh Step 5 [no] ui-idle-timeout-seconds seconds Example: Enables or disables refresh in the session records. The session record flags are specified as a comma-separated value list of one or more of the following flags: login, logout, and refresh. Sets the maximum GUI idle duration before requiring login refresh. The range is 60 to 65525 seconds. apic1(config-webtoken)# ui-idle-timeout-seconds 120 Step 6 [no] webtoken-timeout-seconds seconds Sets the webtoken timeout interval. The range is 600 to 9600 seconds. Example: apic1(config-webtoken)# webtoken-timeout-seconds 1200 Step 7 exit Example: apic1(config-webtoken)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 287 Configuring Security Configuring Communication Policies Examples This example shows how to configure a webtoken. apic1# configure apic1(config)# crypto webtoken apic1(config-webtoken)# max-validity-period 10 apic1(config-webtoken)# session-record-flags login,refresh apic1(config-webtoken)# ui-idle-timeout-seconds 120 apic1(config-webtoken)# webtoken-timeout-seconds 1200 Configuring Communication Policies Configuring the HTTP Policy Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] comm-policy {default | policy-name} Enters communication policy configuration mode. Example: apic1(config)# comm-policy myCommPolicy Step 3 http Enters HTTP policy configuration mode. Example: apic1(config-comm-policy)# http Step 4 [no] admin-state-enable Enables HTTP communication service. Example: apic1(config-http)# admin-state-enable Step 5 [no] allow-origin url Specifies the URL to return in the Access-Control-Allow-Origin HTTP header. Example: apic1(config-http)# allow-origin www.example.com Step 6 [no] port port-number Example: apic1(config-http)# port 8080 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 288 Sets the port used for HTTP communication service. Configuring Security Configuring the HTTPS Policy Step 7 Command or Action Purpose [no] redirect Enables HTTP redirection. Example: apic1(config-http)# no redirect Step 8 [no] request-status-count count Sets the maximum count of HTTP requests to track. The range is 0 to 10240. Example: apic1(config-http)# request-status-count 512 Step 9 Returns to communications policy configuration mode. exit Example: apic1(config-http)# exit Examples This example shows how to configure HTTP service. apic1# configure apic1(config)# comm-policy myCommPolicy apic1(config-comm-policy)# http apic1(config-http)# admin-state-enable apic1(config-http)# allow-origin www.example.com apic1(config-http)# port 8080 apic1(config-http)# no redirect apic1(config-http)# request-status-count 512 apic1(config-http)# exit Configuring the HTTPS Policy Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] comm-policy {default | policy-name} Enters communication policy configuration mode. Example: apic1(config)# comm-policy myCommPolicy Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 289 Configuring Security Configuring the HTTPS Policy Step 3 Command or Action Purpose https Enters HTTPS policy configuration mode. Example: apic1(config-comm-policy)# https Step 4 [no] admin-state-enable Enables HTTPS communication service. Example: apic1(config-https)# admin-state-enable Step 5 [no] port port-number Sets the port used for HTTPS communication service. Example: apic1(config-https)# port 443 Step 6 [no] request-status-count count Sets the maximum count of HTTPS requests to track. The range is 0 to 10240. Example: apic1(config-https)# request-status-count 512 Step 7 [no] ssl-protocols {TLSv1 | TLSv1.1 | TLSv1.2} Specifies in a comma-separated list the SSL protocols that are supported. The options are TLSv1, TLSv1.1, and TLSv1.2. Example: apic1(config-https)# ssl-protocols TLSv1.1,TLSv1.2 Step 8 [no] use-keyring keyring-name Specifies a keyring to use for the HTTPS server SSL certificate. Example: apic1(config-https)# use-keyring myKeyRing Step 9 exit Example: apic1(config-https)# exit Examples This example shows how to configure HTTPS service. apic1# configure apic1(config)# comm-policy myCommPolicy apic1(config-comm-policy)# https apic1(config-https)# admin-state-enable apic1(config-https)# port 443 apic1(config-https)# request-status-count 512 apic1(config-https)# ssl-protocols TLSv1.1,TLSv1.2 apic1(config-https)# use-keyring myKeyRing apic1(config-https)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 290 Returns to communications policy configuration mode. Configuring Security Configuring the SSH Policy Configuring the SSH Policy Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] comm-policy {default | policy-name} Enters communication policy configuration mode. Example: apic1(config)# comm-policy myCommPolicy Step 3 Enters SSH policy configuration mode. ssh-service Example: apic1(comm-policy)# ssh-service Step 4 [no] admin-state-enable Enables HTTP communication service. Example: apic1(config-ssh-service)# admin-state-enable Step 5 [no] port port-number Sets the port used for SSH communication service. Example: apic1(config-ssh-service)# port 22 Step 6 Returns to communications policy configuration mode. exit Example: apic1(config-ssh-service)# exit Examples This example shows how to configure SSH service. apic1# configure apic1(config)# comm-policy apic1(config-comm-policy)# apic1(config-ssh-service)# apic1(config-ssh-service)# apic1(config-ssh-service)# myCommPolicy ssh-service admin-state-enable port 22 exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 291 Configuring Security Configuring the Telnet Policy Configuring the Telnet Policy Before You Begin To allow telnet communications, you must configure an out-of-band contract allowing telnet traffic, which is normally on TCP and UDP ports 23. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] comm-policy {default | policy-name} Enters communication policy configuration mode. Example: apic1(config)# comm-policy myCommPolicy Step 3 telnet Enters Telnet policy configuration mode. Example: apic1(config-comm-policy)# telnet Step 4 [no] admin-state-enable Enables Telnet communication service. Example: apic1(config-telnet)# admin-state-enable Step 5 [no] port port-number Sets the port used for Telnet communication service. Example: apic1(config-telnet)# port 23 Step 6 exit Example: apic1(config-telnet)# exit Examples This example shows how to configure Telnet service. apic1# configure apic1(config)# comm-policy myCommPolicy apic1(config-comm-policy)# telnet apic1(config-telnet)# admin-state-enable apic1(config-telnet)# port 23 apic1(config-telnet)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 292 Returns to communications policy configuration mode. Configuring Security Configuring AES Encryption Configuring AES Encryption Beginning with Cisco APIC Release 1.1(2), the secure properties of APIC configuration files can be encrypted by enabling AES-256 encryption. AES encryption is a global configuration option; all secure properties conform to the AES configuration setting. It is not possible to export a subset of the ACI fabric configuration such as a tenant configuration with AES encryption while not encrypting the remainder of the fabric configuration. For a list of secure properties, see "Appendix K: Secure Properties" in Cisco Application Centric Infrastructure Fundamentals. The APIC uses a 16 to 32 character passphrase to generate the AES-256 keys. The APIC GUI displays a hash of the AES passphrase. This hash can be used to see whether the same passphrase is used on two ACI fabrics. This hash can be copied to a client computer where it can be compared to the passphrase hash of another ACI fabric to see if they were generated with the same passphrase. The hash cannot be used to reconstruct the original passphrase or the AES-256 keys. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 Enters AES configuration mode. crypto aes Example: apic1(config)# crypto aes Step 3 (Optional) Deletes any existing AES encryption key. clear-encryption-key Example: apic1(config-aes)# clear-encryption-key Step 4 passphrase Example: apic1(config-aes)# passphrase Enter passphrase: "This is my passphrase" Enter passphrase again: "This is my passphrase" Step 5 Specifies the AES encryption passphrase. The passphrase can be 16 to 32 characters and must be enclosed in quotes. For increased security, entering the passphrase is interactive. Enables (or disables) AES encryption. [no] encryption Example: apic1(config-aes)# encryption Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 293 Configuring Security Configuring Fabric Secure Mode Examples This example shows how to enable AES encryption and configure a passphrase. apic1# configure apic1(config)# crypto aes apic1(config-aes)# clear-encryption-key apic1(config-aes)# passphrase "This is my passphrase" apic1(config-aes)# encryption Configuring Fabric Secure Mode Fabric secure mode prevents parties with physical access to the fabric equipment from adding a switch or APIC controller to the fabric without manual authorization by an administrator. Starting with Cisco APIC Release 1.2(1x), the firmware checks that switches and controllers in the fabric have valid serial numbers associated with a valid Cisco digitally signed certificate. This validation is performed upon upgrade to this release or during an initial installation of the fabric. The default setting for this feature is permissive mode; an existing fabric continues to run as it has after an upgrade to Release 1.2(1). An administrator with fabric-wide access rights must enable strict mode. Permissive Mode (default) operates as follows: • Allows an existing fabric to operate normally even though one or more switches have an invalid certificate. • Does not enforce serial number based authorization. • Allows auto-discovered controllers and switches to join the fabric without enforcing serial number authorization. Strict Mode operates as follows: • Only switches with a valid Cisco serial number and SSL certificate are allowed. • Enforces serial number based authorization. • Requires an administrator to manually authorize controllers and switches to join the fabric. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 system fabric-security-mode {permissive | strict} Example: apic1(config)# system fabric-security-mode strict Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 294 Specifies the fabric security mode. Configuring Security Configuring COOP Authentication Command or Action Step 3 Purpose system controller-id controller-id {approve | reject} In strict mode, approves or rejects a controller to join the fabric. Example: apic1(config)# system controller-id FCH1750V025 approve Examples This example shows how to change the fabric security mode to strict. apic1# configure apic1(config)# system fabric-security-mode strict This example shows how to approve a controller to join the fabric when strict mode is configured. apic1# configure apic1(config)# system controller-id FCH1750V025 approve Configuring COOP Authentication About COOP Authentication Council of Oracles Protocol (COOP) is used to communicate the mapping information (location and identity) to the spine proxy. A leaf switch will forward endpoint address information to a spine using ZeroMQ (Zero Message Queue or ZMQ). COOP running on the spine nodes ensures that all spine nodes maintain a consistent copy of end point address and location information and additionally maintains the distributed hash table (DHT) repository of endpoint identity to location mapping database. Without COOP authentication, it is possible for users to send arbitrary COOP messages, which would be acted on by the fabric nodes. Cisco APIC Release 2.0 adds an MD5 TCP option to provide authentication and integrity protection to the ZMQ TCP transportation. Two authentication modes are supported: • Compatible - COOP accepts both MD5 authenticated and non-authenticated ZMQ connections for message transportation. COOP data path communication gives high priority to transport via secured connections. • Strict - COOP allows MD5 authenticated ZMQ connections only. Changing the configuration of the COOP authentication type has the following effects: • When the configuration changes from compatible to strict mode, all non-authenticated ZMQ connections are disconnected. • When the configuration changes from strict to compatible mode, COOP immediately accepts both authenticated and non-authenticated ZMQ connections. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 295 Configuring Security Configuring COOP Authentication Configuring COOP Authentication Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 coop-fabric Enters COOP fabric configuration mode. Example: apic1(config)# coop-fabric Step 3 authentication type {compatible | strict} Configures the COOP authentication type as one of the following: Example: apic1(config-coop-fabric)# authentication type compatible • compatible - COOP allows MD5 authenticated and non-authenticated ZMQ connections. • strict - allows MD5 authenticated ZMQ connections only. This example shows how to configure COOP authentication in compatible mode: apic1# configure apic1(config)# coop-fabric apic1(config-coop-fabric# authentication type compatible Configuring FIPS About Federal Information Processing Standards (FIPS) The Federal Information Processing Standards (FIPS) Publication 140-2, Security Requirements for Cryptographic Modules, details the U.S. government requirements for cryptographic modules. FIPS 140-2 specifies that a cryptographic module should be a set of hardware, software, firmware, or some combination that implements cryptographic functions or processes, including cryptographic algorithms and, optionally, key generation, and is contained within a defined cryptographic boundary. FIPS specifies certain cryptographic algorithms as secure, and it also identifies which algorithms should be used if a cryptographic module is to be called FIPS compliant. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 296 Configuring Security Guidelines and Limitations Guidelines and Limitations Follow these guidelines and limitations: • When FIPS is enabled, it is applied across Cisco APIC. • When performing a Cisco APIC software downgrade, you must disable FIPS first. • Make your passwords a minimum of eight characters in length. • Disable Telnet. Users should log in using SSH only. • Delete all SSH Server RSA1 keypairs. • Disable remote authentication through RADIUS/TACACS+. Only local and LDAP users can be authenticated. • Secure Shell (SSH) and SNMP are supported. • Disable SNMP v1 and v2. Any existing user accounts on the switch that have been configured for SNMPv3 should be configured only with SHA for authentication and AES/3DES for privacy. • Starting with release 2.3(1x), FIPS can be configured at the switch level. Configuring FIPS for Cisco APIC Using NX-OS Style CLI When FIPS is enabled, it is applied across Cisco APIC. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 fips mode enable Enables FIP. The no fips mode enable command disables FIPS. Example: You must reboot to complete the configuration. Anytime you change the mode, you must reboot to complete the configuration. apic1(config)# fips mode enable Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 297 Configuring Security Configuring Control Plane Policing Configuring Control Plane Policing Information About CoPP Control Plane Policing (CoPP) protects the control plane, which ensures network stability, reachability, and packet delivery. This feature allows specification of parameters, for each protocol that can reach the control processor to be rate-limited using a policer. The policing is applied to all traffic destined to any of the IP addresses of the router or Layer 3 switch. A common attack vector for network devices is the denial-of-service (DoS) attack, where excessive traffic is directed at the device interfaces. The Cisco ACI Leaf/Spine NX-OS provides CoPP to prevent DoS attacks from impacting performance. Such attacks, which can be perpetrated either inadvertently or maliciously, typically involve high rates of traffic destined to the supervisor module of an ACI Leaf/Spine CPU or CPU itself. The supervisor module of ACI Leaf/Spine switches divides the traffic that it manages into two functional components or planes: • Data plane—Handles all the data traffic. The basic functionality of a Cisco NX-OS device is to forward packets from one interface to another. The packets that are not meant for the switch itself are called the transit packets. These packets are handled by the data plane. • Control plane—Handles all routing protocol control traffic. These protocols, such as the Border Gateway Protocol (BGP) and the Open Shortest Path First (OSPF) Protocol, send control packets between devices. These packets are destined to router addresses and are called control plane packets. The ACI Leaf/Spine supervisor module has a control plane and is critical to the operation of the network. Any disruption or attacks to the supervisor module will result in serious network outages. For example, excessive traffic to the supervisor module could overload and slow down the performance of the entire Cisco ACI fabric. Another example is a DoS attack on the ACI Leaf/Spine supervisor module that could generate IP traffic streams to the control plane at a very high rate, forcing the control plane to spend a large amount of time in handling these packets and preventing the control plane from processing genuine traffic. Examples of DoS attacks are as follows: • Internet Control Message Protocol (ICMP) echo requests • IP fragments • TCP SYN flooding These attacks can impact the device performance and have the following negative effects: • Reduced service quality (such as poor voice, video, or critical applications traffic) • High route processor or switch processor CPU utilization • Route flaps due to loss of routing protocol updates or keepalives • Processor resource exhaustion, such as the memory and buffers • Indiscriminate drops of incoming packets Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 298 Configuring Security Information About CoPP Note ACI Leaf/Spines are by default protected by CoPP with default settings. This feature allows for tuning the parameters on a group of nodes based on customer needs. Control Plane Protection To protect the control plane, the Cisco NX-OS running on ACI Leaf/Spines segregates different packets destined for the control plane into different classes. Once these classes are identified, the Cisco NX-OS device polices the packets, which ensures that the supervisor module is not overwhelmed. Control Plane Packet Types: Different types of packets can reach the control plane: • Receive Packets—Packets that have the destination address of a router. The destination address can be a Layer 2 address (such as a router MAC address) or a Layer 3 address (such as the IP address of a router interface). These packets include router updates and keepalive messages. Multicast packets can also be in this category where packets are sent to multicast addresses that are used by a router. • Exception Packets—Packets that need special handling by the supervisor module. For example, if a destination address is not present in the Forwarding Information Base (FIB) and results in a miss, the supervisor module sends an ICMP unreachable packet back to the sender. Another example is a packet with IP options set. • Redirect Packets—Packets that are redirected to the supervisor module. Features such as Dynamic Host Configuration Protocol (DHCP) snooping or dynamic Address Resolution Protocol (ARP) inspection redirect some packets to the supervisor module. • Glean Packets—If a Layer 2 MAC address for a destination IP address is not present in the FIB, the supervisor module receives the packet and sends an ARP request to the host. All of these different packets could be maliciously used to attack the control plane and overwhelm the Cisco ACI Fabric. CoPP classifies these packets to different classes and provides a mechanism to individually control the rate at which the ACI Leaf/Spine supervisor module receives these packets. Classification for CoPP: For effective protection, the ACI Leaf/Spine NX-OS classifies the packets that reach the supervisor modules to allow you to apply different rate controlling policies based on the type of the packet. For example, you might want to be less strict with a protocol packet such as Hello messages but more strict with a packet that is sent to the supervisor module because the IP option is set. Rate Controlling Mechanisms: Once the packets are classified, the ACI Leaf/Spine NX-OS has different mechanisms to control the rate at which packets arrive at the supervisor module. You can configure the following parameters for policing: • Committed information rate (CIR)—Desired bandwidth, specified as a bit rate or a percentage of the link rate. • Committed burst (BC)—Size of a traffic burst that can exceed the CIR within a given unit of time and not impact scheduling. Default Policing Policies: Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 299 Configuring Security Guidelines and Limitations for CoPP When the Cisco ACI Leaf/Spine is bootup, the platform setup pre-defined CoPP parameters for different protocols are based on the tests done by Cisco. Guidelines and Limitations for CoPP CoPP has the following configuration guidelines and limitations: • We recommend that you use the strict default CoPP policy initially and then later modify the CoPP policies based on the data center and application requirements. • Customizing CoPP is an ongoing process. CoPP must be configured according to the protocols and features used in your specific environment as well as the supervisor features that are required by the server environment. As these protocols and features change, CoPP must be modified. • We recommend that you continuously monitor CoPP. If drops occur, determine if CoPP dropped traffic unintentionally or in response to a malfunction or attack. In either event, analyze the situation and evaluate the need to modify the CoPP policies. • You must ensure that the CoPP policy does not filter critical traffic such as routing protocols or interactive access to the device. Filtering this traffic could prevent remote access to the Cisco ACI Leaf/Spine and require a console connection. • You can use the APIC UI to be able to tune the CoPP parameters. Configuring CoPP Using the Cisco NX-OS CLI Procedure Step 1 Configure a CoPP leaf profile: Example: # configure copp Leaf Profile apic1(config)# policy-map type control-plane-leaf leafProfile apic1(config-pmap-copp-leaf)# profile-type custom apic1(config-pmap-copp-leaf)# set arpRate 786 # create a policy group to be applied on leaves apic1(config)# template leaf-policy-group coppForLeaves apic1(config-leaf-policy-group)# copp-aggr leafProfile apic1(config-leaf-policy-group)# exit # apply the leaves policy group on leaves apic1(config)# leaf-profile applyCopp apic1(config-leaf-profile)# leaf-group applyCopp apic1(config-leaf-group)# leaf 101-102 apic1(config-leaf-group)# leaf-policy-group coppForLeaves Step 2 Configure a CoPP Spine profile: Example: # configure copp Spine Profile apic1(config)# policy-map type control-plane-spine spineProfile apic1(config-pmap-copp-spine)# profile-type custom apic1(config-pmap-copp-spine)# set arpRate 786 # create a policy group to be applied on spines apic1(config)# template leaf-policy-group coppForSpines apic1(config-spine-policy-group)# copp-aggr spineProfile Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 300 Configuring Security Configuring First Hop Security apic1(config-spine-policy-group)# exit # apply the spine policy group on spines apic1(config)# spine-profile applyCopp apic1(config-spine-profile)# spine-group applyCopp apic1(config-spine-group)# spine 201-202 apic1(config-spine-group)# spine-policy-group coppForSpines Configuring First Hop Security About First Hop Security First-Hop Security (FHS) features enable a better IPv4 and IPv6 link security and management over the layer 2 links. In a service provider environment, these features closely control address assignment and derived operations, such as Duplicate Address Detection (DAD) and Address Resolution (AR). The following supported FHS features secure the protocols and help build a secure endpoint database on the fabric leaf switches, that are used to mitigate security threats such as MIM attacks and IP thefts: • ARP Inspection • ND Inspection • DHCP Inspection • RA Guard • IPv4 and IPv6 Source Guard • Trust Control FHS features provide the following security measures: • Role Enforcement—Prevents untrusted hosts from sending messages that are out the scope of their role. • Binding Enforcement—Prevents address theft. • DoS Attack Mitigations—Prevents malicious end-points to grow the end-point database to the point where the database could stop providing operation services. • Proxy Services—Provides some proxy-services to increase the efficiency of address resolution. FHS features are enabled on a per tenant bridge domain (BD) basis. As the bridge domain, may be deployed on a single or across multiple leaf switches, the FHS threat control and mitigation mechanisms cater to a single switch and multiple switch scenarios. ACI FHS Deployment Most FHS features are configured in a two-step fashion: firstly you define a policy which describes the behavior of the feature, secondly you apply this policy to a "domain" (being the Tenant Bridge Domain or the Tenant Endpoint Group). Different policies that define different behaviors can be applied to different intersecting Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 301 Configuring Security Guidelines and Limitations domains. The decision to use a specific policy is taken by the most specific domain to which the policy is applied. The policy options can be defined from the Cisco APIC GUI found under the Tenant_name>Networking>Protocol Policies>First Hop Security tab. Guidelines and Limitations Follow these guidelines and limitations: • FHS is not supported for AVS VXLAN domain. • FHS feature is not supported when an EPG is deployed with VXLAN encapsulation. • Any secured endpoint entry in the FHS Binding Table Database in DOWN state will get cleared after 18 Hours of timeout. The entry moves to DOWN state when the front panel port where the entry is learned is link down. During this window of 18 Hours, if the endpoint is moved to a different location and is seen on a different port, the entry will be gracefully moved out of DOWN state to REACHABLE/STALE as long as the endpoint is reachable from the other port it is moved from. • When IP Source Guard is enabled, the IPv6 traffic that is sourced using IPv6 Link Local address as IP source address is not subject to the IP Source Guard enforcement (i.e. Enforcement of Source Mac <=> Source IP Bindings secured by IP Inspect Feature). This traffic is permitted by default irrespective of binding check failures. • FHS is not supported on L3Out interfaces. • FHS is not supported N9K-M12PQ based TORs. • FHS is not supported for MultiSite environment. • FHS is not supported on a Layer 2 only bridge domain. Configuring FHS Using the NX-OS CLI Before You Begin • The tenant and Bridge Domain configured. Procedure Step 1 configure Enters configuration mode. Example: apic1# configure Step 2 Configure FHS policy. Example: apic1(config)# tenant coke apic1(config-tenant)# first-hop-security Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 302 Configuring Security Configuring FHS Using the NX-OS CLI apic1(config-tenant-fhs)# security-policy pol1 apic1(config-tenant-fhs-secpol)# apic1(config-tenant-fhs-secpol)# ip-inspection-admin-status enabled-both apic1(config-tenant-fhs-secpol)# source-guard-admin-status enabled-both apic1(config-tenant-fhs-secpol)# router-advertisement-guard-admin-status enabled apic1(config-tenant-fhs-secpol)# router-advertisement-guard apic1(config-tenant-fhs-raguard)# apic1(config-tenant-fhs-raguard)# managed-config-check apic1(config-tenant-fhs-raguard)# managed-config-flag apic1(config-tenant-fhs-raguard)# other-config-check apic1(config-tenant-fhs-raguard)# other-config-flag apic1(config-tenant-fhs-raguard)# maximum-router-preference low apic1(config-tenant-fhs-raguard)# minimum-hop-limit 10 apic1(config-tenant-fhs-raguard)# maximum-hop-limit 100 apic1(config-tenant-fhs-raguard)# exit apic1(config-tenant-fhs-secpol)# exit apic1(config-tenant-fhs)# trust-control tcpol1 pic1(config-tenant-fhs-trustctrl)# arp apic1(config-tenant-fhs-trustctrl)# dhcpv4-server apic1(config-tenant-fhs-trustctrl)# dhcpv6-server apic1(config-tenant-fhs-trustctrl)# ipv6-router apic1(config-tenant-fhs-trustctrl)# router-advertisement apic1(config-tenant-fhs-trustctrl)# neighbor-discovery apic1(config-tenant-fhs-trustctrl)# exit apic1(config-tenant-fhs)# exit apic1(config-tenant)# bridge-domain bd1 apic1(config-tenant-bd)# first-hop-security security-policy pol1 apic1(config-tenant-bd)# exit apic1(config-tenant)# application ap1 apic1(config-tenant-app)# epg epg1 apic1(config-tenant-app-epg)# first-hop-security trust-control tcpol1 Step 3 Show FHS configuration example: Example: leaf4# show fhs bt all Legend: TR : trusted-access : Age since creation UNTR : untrusted-access : creating UNKNW : unknown : invalid NDP : Neighbor Discovery Protocol : reachable INCMP : incomplete : Interface TimeLeft : Remaining time since last refresh : dhcp-assigned EPG-Mode: U : unknown BD-VNID 15630220 M : mac BD-Vlan 3 V : vlan UNRES : unresolved Age UNDTR : undetermined-trust CRTNG TENTV : tentative INV STA REACH : static-authenticated VERFY : verify INTF LM DHCP : lla-mac-match I : ip BD-Name t0:bd200 --------------------------------------------------------------------------------------------------------------------| Origin | IP | MAC | INTF | EPG(sclass)(mode) | Trust-lvl | State | Age | TimeLeft | --------------------------------------------------------------------------------------------------------------------| ARP | 192.0.200.12 | D0:72:DC:A0:3D:4F | eth1/1 | epg300(49154)(V) | LM,TR | STALE | 00:04:49 | 18:08:13 | | ARP | 172.29.205.232 | D0:72:DC:A0:3D:4F | eth1/1 | epg300(49154)(V) | LM,TR | STALE | 00:03:55 | 18:08:21 | | ARP | 192.0.200.21 | D0:72:DC:A0:3D:4F | eth1/1 | epg300(49154)(V) | LM,TR | REACH | 00:03:36 | 00:00:02 | | LOCAL | 192.0.200.1 | 00:22:BD:F8:19:FF | vlan3 | LOCAL(16387)(I) | STA | REACH | 04:49:41 | N/A | | LOCAL | fe80::200 | 00:22:BD:F8:19:FF | vlan3 | LOCAL(16387)(I) | STA | Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 303 Configuring Security Configuring FHS Using the NX-OS CLI REACH | 04:49:40 | N/A | | LOCAL | 2001:0:0:200::1 | 00:22:BD:F8:19:FF | vlan3 | LOCAL(16387)(I) | STA | REACH | 04:49:39 | N/A | --------------------------------------------------------------------------------------------------------------------- Step 4 Show violations with the different types and reasons example: Example: leaf4# show fhs violations all Violation-Type: POL : policy ROLE : role INT : internal THR : address-theft-remote TH : address-theft Violation-Reason: IP-MAC-TH : ip-mac-theft ANC-COL : anchor-collision PRF-LVL-CHK : ra-rtr-pref-level-check-fail TRUST-CHK : trust-check-fail SRV-ROL-CHK : srv-role-check-fail LCL-EP-COL : local-ep-collision MAC-TH : mac-theft MCFG-CHK : ra-managed-cfg-check-fail HOP-LMT-CHK : ra-hoplimit-check-fail RTR-ROL-CHK : rtr-role-check-fail IP-TH : ip-theft EPG-Mode: U : unknown M : mac V : vlan OCFG_CHK : ra-other-cfg-check-fail INT-ERR : internal-error ST-EP-COL : static-ep-collision EP-LIM : ep-limit-reached MOV-COL : competing-move-collision I : ip BD-VNID BD-Vlan BD-Name 15630220 3 t0:bd200 ----------------------------------------------------------------------------------------------------| Type | Last-Reason | Proto | IP | MAC | Port | EPG(sclass)(mode) | Count | ----------------------------------------------------------------------------------------------------| THR | IP-TH | ARP | 192.0.200.21 | D0:72:DC:A0:3D:4F | tunnel5 | epg300(49154)(V) | 21 | ----------------------------------------------------------------------------------------------------Table Count: 1 Step 5 Show FHS configuration: Example: swtb23-ifc1# show tenant t0 bridge-domain bd200 first-hop-security binding-table Pod/Node Type Family State -------- ------ -------------------- ----1/102 local ipv4 reach IP Address MAC Address Interface Level -------------------- ----------------- ------------ 192.0.200.1 00:22:BD:F8:19:FF vlan3 static- authenticated 1/102 local reach able ipv6 fe80::200 00:22:BD:F8:19:FF vlan3 static- authenticated 1/102 local reach able ipv6 2001:0:0:200::1 00:22:BD:F8:19:FF vlan3 static- authenticated 1/101 arp stale able ipv4 192.0.200.23 D0:72:DC:A0:02:61 eth1/2 lla-mac-match ,untrusted- 1/101 local reach ipv4 192.0.200.1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 304 00:22:BD:F8:19:FF vlan3 access static- Configuring Security Configuring FHS Using the NX-OS CLI authenticated 1/101 nd reach able ipv6 fe80::d272:dcff:fea0 D0:72:DC:A0:02:61 eth1/2 lla-mac-match :261 ,untrusted- able 1/101 nd stale ipv6 2001:0:0:200::20 D0:72:DC:A0:02:61 access lla-mac-match eth1/2 ,untrusted1/101 nd stale ipv6 2001::200:d272:dcff: D0:72:DC:A0:02:61 access lla-mac-match eth1/2 fea0:261 ,untrusted- fe80::200 00:22:BD:F8:19:FF vlan3 access static- able ipv6 2001:0:0:200::1 00:22:BD:F8:19:FF vlan3 static- authenticated 1/103 local reach able ipv4 192.0.200.1 00:22:BD:F8:19:FF vlan4 static- authenticated 1/103 local reach able ipv6 fe80::200 00:22:BD:F8:19:FF vlan4 static- authenticated 1/103 local reach able ipv6 2001:0:0:200::1 00:22:BD:F8:19:FF vlan4 static- authenticated 1/104 arp stale able ipv4 192.0.200.10 F8:72:EA:AD:C4:7C eth1/1 lla-mac-match ipv4 172.29.207.222 D0:72:DC:A0:3D:4C eth1/1 lla-mac-match 1/101 local reach ipv6 authenticated 1/101 local reach ,trusted-access 1/104 arp stale ,trusted-access 1/104 local reach authenticated 1/104 nd stale ipv4 able ipv6 192.0.200.1 fe80::fa72:eaff:fead 00:22:BD:F8:19:FF vlan4 static- F8:72:EA:AD:C4:7C eth1/1 lla-mac-match F8:72:EA:AD:C4:7C eth1/1 lla-mac-match :c47c ,trusted-access 1/104 nd ipv6 stale ,trusted-access 1/104 local reach ipv6 authenticated 1/104 local reach able ipv6 authenticated able Pod/Node 2001:0:0:200::10 fe80::200 00:22:BD:F8:19:FF vlan4 static- 2001:0:0:200::1 00:22:BD:F8:19:FF vlan4 static- Type IP Address Creation TS Lease Period -------- ------ -------------------- --------------------------------------------------------- -----------1/102 local 192.0.200.1 2017-07-20T04:22:38.000+00:00 Last Refresh TS Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 305 Configuring Security Configuring FHS Using the NX-OS CLI 2017-07-20T04:22:38.000+00:00 1/102 local fe80::200 2017-07-20T04:22:56.000+00:00 1/102 local 2001:0:0:200::1 2017-07-20T04:22:57.000+00:00 1/101 arp 192.0.200.23 2017-07-27T16:07:24.000+00:00 1/101 local 192.0.200.1 2017-07-27T10:48:09.000+00:00 1/101 nd fe80::d272:dcff:fea0 2017-07-27T16:04:29.000+00:00 :261 1/101 nd 2001:0:0:200::20 2017-07-27T16:07:24.000+00:00 1/101 nd 2001::200:d272:dcff: 2017-07-27T16:07:24.000+00:00 fea0:261 1/101 local fe80::200 2017-07-27T10:48:10.000+00:00 1/101 local 2001:0:0:200::1 2017-07-27T10:48:11.000+00:00 1/103 local 192.0.200.1 2017-07-26T22:03:56.000+00:00 1/103 local fe80::200 2017-07-26T22:03:57.000+00:00 1/103 local 2001:0:0:200::1 2017-07-26T22:03:58.000+00:00 1/104 arp 192.0.200.10 2017-07-27T16:05:48.000+00:00 1/104 arp 172.29.207.222 2017-07-27T16:06:38.000+00:00 1/104 local 192.0.200.1 2017-07-27T10:49:13.000+00:00 1/104 nd fe80::fa72:eaff:fead 2017-07-27T16:06:43.000+00:00 :c47c 1/104 nd 2001:0:0:200::10 2017-07-27T16:06:19.000+00:00 1/104 local fe80::200 2017-07-27T10:49:14.000+00:00 1/104 local 2001:0:0:200::1 2017-07-27T10:49:15.000+00:00 2017-07-20T04:22:56.000+00:00 2017-07-20T04:22:57.000+00:00 2017-07-27T10:55:20.000+00:00 2017-07-27T10:48:09.000+00:00 2017-07-27T10:52:16.000+00:00 2017-07-27T10:57:32.000+00:00 2017-07-27T11:21:45.000+00:00 2017-07-27T10:48:10.000+00:00 2017-07-27T10:48:11.000+00:00 2017-07-26T22:03:56.000+00:00 2017-07-26T22:03:57.000+00:00 2017-07-26T22:03:58.000+00:00 2017-07-27T11:21:13.000+00:00 2017-07-27T11:54:48.000+00:00 2017-07-27T10:49:13.000+00:00 2017-07-27T11:21:13.000+00:00 2017-07-27T11:21:13.000+00:00 2017-07-27T10:49:14.000+00:00 2017-07-27T10:49:15.000+00:00 swtb23-ifc1# swtb23-ifc1# show tenant t0 bridge-domain bd200 first-hop-security statistics arp Pod/Node : 1/101 Request Received : 4 Request Switched : 2 Request Dropped : 2 Reply Received : 257 Reply Switched : 257 Reply Dropped : 0 Pod/Node Request Received Request Switched Request Dropped Reply Received Reply Switched Reply Dropped : : : : : : : 1/104 6 6 0 954 954 0 swtb23-ifc1# show tenant t0 bridge-domain bd200 first-hop-security statistics dhcpv4 Pod/Node : 1/102 Discovery Received : 5 Discovery Switched : 5 Discovery Dropped : 0 Offer Received : 0 Offer Switched : 0 Offer Dropped : 0 Request Received : 0 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 306 Configuring Security Configuring FHS Using the NX-OS CLI Request Switched Request Dropped Ack Received Ack Switched Ack Dropped Nack Received Nack Switched Nack Dropped Decline Received Decline Switched Decline Dropped Release Received Release Switched Release Dropped Information Received Information Switched Information Dropped Lease Query Received Lease Query Switched Lease Query Dropped Lease Active Received Lease Active Switched Lease Active Dropped Lease Unassignment Received Lease Unassignment Switched Lease Unassignment Dropped Lease Unknown Received Lease Unknown Switched Lease Unknown Dropped : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 swtb23-ifc1# show tenant t0 bridge-domain bd200 first-hop-security statistics neighbor-discovery Pod/Node : 1/101 Neighbor Solicitation Received : 125 Neighbor Solicitation Switched : 121 Neighbor Solicitation Dropped : 4 Neighbor Advertisement Received : 519 Neighbor Advertisement Switched : 519 Neighbor Advertisement Drop : 0 Router Solicitation Received : 4 Router Solicitation Switched : 4 Router Solicitation Dropped : 0 Router Adv Received : 0 Router Adv Switched : 0 Router Adv Dropped : 0 Redirect Received : 0 Redirect Switched : 0 Redirect Dropped : 0 Pod/Node Neighbor Solicitation Received Neighbor Solicitation Switched Neighbor Solicitation Dropped Neighbor Advertisement Received Neighbor Advertisement Switched Neighbor Advertisement Drop Router Solicitation Received Router Solicitation Switched Router Solicitation Dropped Router Adv Received Router Adv Switched Router Adv Dropped Redirect Received Redirect Switched Redirect Dropped : : : : : : : : : : : : : : : : 1/104 123 47 76 252 228 24 0 0 0 53 6 47 0 0 0 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 307 Configuring Security Configuring FHS Using the NX-OS CLI Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 308 CHAPTER 9 Configuring VMM For information about configuring virtual machine management using the NX-OS style CLI for APIC, see the Cisco ACI Virtualization Guide. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 309 Configuring VMM Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 310 CHAPTER 10 Configuring Layer 4 - Layer 7 Services For information about configuring Layer 4 - Layer 7 services using the NX-OS style CLI for APIC, see the Cisco APIC Layer 4 to Layer 7 Services Deployment Guide. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 311 Configuring Layer 4 - Layer 7 Services Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 312 CHAPTER 11 Configuring Global Policies • About Global Policies, page 313 • Configuring Out-of-Band Management NTP, page 314 • Configuring the System Clock, page 316 • Configuring Error Disable Recovery, page 317 • Configuring Link Level Discovery Protocol, page 318 • Configuring Miscabling Protocol, page 318 • Configuring the Endpoint Loop Protection Policy, page 320 • Configuring IP Aging, page 321 • Configuring the Dynamic Load Balancer, page 321 • Configuring Spanning Tree Protocol, page 322 • Configuring IS-IS, page 324 • Configuring BGP Route Reflectors, page 326 • Decommissioning a Node, page 327 • Configuring Power Management, page 328 • Configuring a Scheduler, page 329 • Configuring System MTU, page 332 • About PTP, page 332 About Global Policies The APIC fabric has many fabric level configurations, which are applied to the entire fabric components (switches and ports). In some cases, lower level policies (switch or interface level) exist to override these policies. For example, while MCP policy can enable the MCP feature globally, an interface level MCP policy exists to enable or disable MCP on an individual interface. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 313 Configuring Global Policies Configuring Out-of-Band Management NTP Configuring Out-of-Band Management NTP When an ACI fabric is deployed with out-of-band management, each node of the fabric is managed from outside the ACI fabric. You can configure an out-of-band management NTP server so that each node can individually query the same NTP server as a consistent clock source. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: apic1# configure Step 2 template ntp-fabric ntp-fabric-template-name Specifies the NTP template (policy) for the fabric. Example: apic1(config)# template ntp-fabric pol1 Step 3 [no] server dns-name-or-ipaddress [prefer] [use-vrf {inband-mgmt | oob-default}] [key key-value] Example: apic1(config-template-ntp-fabric)# server 192.0.20.123 prefer use-vrf oob-mgmt Step 4 [no] authenticate Configures an NTP server for the active NTP policy. To make this server the preferred server for the active NTP policy, include the prefer keyword. If NTP authentication is enabled, specify a reference key ID. To specify the in-band or out-of-band management access VRF, include the use-vrf keyword with the inb-default or oob-default keyword. Enables (or disables) NTP authentication. Example: apic1(config-template-ntp-fabric)# no authenticate Step 5 [no] authentication-key key-value Configures an authentication NTP authentication. The range is 1 to 65535. Example: apic1(config-template-ntp-fabric)# authentication-key 12345 Step 6 [no] trusted-key key-value Configures a trusted NTP authentication. The range is 1 to 65535. Example: apic1(config-template-ntp-fabric)# trusted-key 54321 Step 7 exit Example: apic1(config-template-ntp-fabric)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 314 Returns to global configuration mode Configuring Global Policies Configuring Out-of-Band Management NTP Command or Action Step 8 Purpose template pod-group pod-group-template-name Configures a pod-group template (policy). Example: apic1(config)# template pod-group allPods Step 9 inherit ntp-fabric ntp-fabric-template-name Example: Configures the NTP fabric pod-group to use the previously configured NTP fabric template (policy). apic1(config-pod-group)# inherit ntp-fabric pol1 Step 10 Returns to global configuration mode exit Example: apic1(config-template-pod-group)# exit Step 11 pod-profile pod-profile-name Configures a pod profile. Example: apic1(config)# pod-profile all Step 12 pods {pod-range-1-255 | all} Configures a set of pods. Example: apic1(config-pod-profile)# pods all Step 13 inherit pod-group pod-group-name Associates the pod-profile with the previously configured pod group. Example: apic1(config-pod-profile-pods)# inherit pod-group allPods Step 14 Returns to EXEC mode. end Example: apic1(config-pod-profile-pods)# end Examples This example shows how to configure a preferred out-of-band NTP server and how to verify the configuration and deployment. apic1# configure t apic1(config)# template ntp-fabric pol1 apic1(config-template-ntp-fabric)# server 192.0.20.123 use-vrf oob-default apic1(config-template-ntp-fabric)# no authenticate apic1(config-template-ntp-fabric)# authentication-key 12345 apic1(config-template-ntp-fabric)# trusted-key 12345 apic1(config-template-ntp-fabric)# exit apic1(config)# template pod-group allPods apic1(config-pod-group)# inherit ntp-fabric pol1 apic1(config-pod-group)# exit apic1(config)# pod-profile all apic1(config-pod-profile)# pods all apic1(config-pod-profile-pods)# inherit pod-group allPods Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 315 Configuring Global Policies Configuring the System Clock apic1(config-pod-profile-pods)# end apic1# apic1# show ntpq nodeid remote refid st t when poll reach delay offset jitter ---- -u ----27 ----64 ----377 -----76.427 -----0.087 -----0.067 u u 3 3 64 64 377 377 75.932 75.932 0.001 0.001 0.021 0.021 -----1 * -----------192.0.20.123 -----.GPS. 2 3 * * 192.0.20.123 192.0.20.123 .GPS. .GPS. Configuring the System Clock Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] clock display-format {local | utc} Sets the clock date time format to either local or UTC time. Example: apic1(config)# clock display-format local Step 3 [no] clock show-offset enable Example: Enables (or disables) the display of the offfset from UTC. This setting is valid only when the display-format is local. apic1(config)# clock show-offset enable Step 4 [no] clock timezone timezone-code Specifies the local time zone. The default is p0_utc. Example: apic1(config)# clock timezone n420_America-Los_Angeles Step 5 show clock Example: Specifies the delay time for LLDP to initialize on any interface . The range is 1 to 10 seconds; the default is 2 seconds. apic1(config)# show clock Examples This example shows how to configure the system clock for local time in the Los Angeles timezone. apic1# configure terminal apic1(config)# clock display-format local apic1(config)# clock show-offset enable apic1(config)# clock timezone n420_America-Los_Angeles Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 316 Configuring Global Policies Configuring Error Disable Recovery apic1(config)# show clock Time : 20:47:37.038 UTC-08:00 Sun Nov 08 2015 Configuring Error Disable Recovery The error disabled recovery (EDR) policy is a fabric level policy that can enable ports that loop detection and BPDU policies disabled after an interval that the administrator can configure. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] errdisable recovery interval seconds Specifies the interval for an interface to recover from the error-disabled state. The range is from 30 to 65535 seconds Example: apic1(config)# errdisable recovery interval 300 Step 3 [no] errdisable recovery cause {bpduguard | ep-move | mcp-loop} Example: apic1(config)# errdisable recovery cause mcp-loop Specifies a condition under which the interface automatically recovers from the error-disabled state, and the device retries bringing the interface up. The default is disabled. The condition options are: • bpduguard—Enable timer to recover from a BPDU guard error disable. • ep-move—Enable timer to recover from an endpoint move error disable. • mcp-loop—Enable timer to recover from an MCP loop error disable. Examples This example shows how to configure EDR to recover from an MCP loop error disable. apic1# configure terminal apic1(config)# errdisable recovery interval 300 apic1(config)# errdisable recovery cause mcp-loop Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 317 Configuring Global Policies Configuring Link Level Discovery Protocol Configuring Link Level Discovery Protocol The Link Layer Discovery Protocol (LLDP) is a device discovery protocol that allows network devices to advertise information about themselves to other devices on the network. LLDP determines the layer 2 connectivity between switches. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] lldp holdtime seconds Specifies the hold time to be sent in LLDP packets. Example: apic1(config)# lldp holdtime Step 3 [no] lldp holdtime seconds Example: Specifies the hold time to be sent in LLDP packets. The range is 10 to 255 seconds; the default is 120 seconds. apic1(config)# lldp holdtime 120 Step 4 [no] lldp reinit seconds Example: Specifies the delay time for LLDP to initialize on any interface . The range is 1 to 10 seconds; the default is 2 seconds. apic1(config)# lldp reinit 2 Step 5 [no] lldp timer seconds Example: Specifies the transmission frequency seconds of LLDP updates in seconds. The range is 5 to 254 seconds; the default is 30 seconds. apic1(config)# lldp timer 30 Examples This example shows how to configure LLDP. apic1# configure terminal apic1(config)# lldp holdtime 120 apic1(config)# lldp reinit 2 apic1(config)# lldp timer 30 Configuring Miscabling Protocol The ACI fabric provides loop detection policies that can detect loops in Layer 2 network segments that are connected to ACI access ports. The ACI fabric implements the mis-cabling protocol (MCP), a fabric level policy that allows provisioning of MCP parameters as well as determining the port behavior if mis-cabling is Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 318 Configuring Global Policies Configuring Miscabling Protocol detected. MCP works in a complementary manner with STP that is running on external Layer 2 networks, and handles Bridge Protocol Data Unit (BPDU) packets that access ports receive. A fabric administrator provides a key that MCP uses to identify which MCP packets are initiated by the ACI fabric. The administrator can choose how the MCP policies identify loops and how to act upon the loops: syslog only, or disable the port. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] mcp action port-disable Specifies whether a port should be place in a disabled state if mis-cabling is detected. Example: apic1(config)# mcp action port-disable Step 3 [no] mcp enable [key key-value] Example: apic1(config)# mcp enable key 0123456789abcdef Step 4 [no] mcp factor number Example: Allows enabling or disabling of the MCP protocol globally for the entire fabric. A password (key) is required when enabling the policy but not when disabling. Sets the loop detection multiplication factor, which is used while sending MCP packets. The range is 1 to 255. apic1(config)# mcp factor 64 Step 5 [no] mcp init-delay seconds Specifies the initial delay time. The range is 0 to 1800 seconds; the default is 180. Example: apic1(config)# mcp init-delay 180 Step 6 [no] mcp transmit-frequency frequency Sets the frequency of transmission of MCP packets to detect mis-cabling. The range is 2 to 300; the default is 2. Example: apic1(config)# mcp transmit-frequency 2 Examples This example shows how to configure MCP. apic1# configure terminal apic1(config)# mcp action port-disable apic1(config)# mcp enable key 0123456789abcdef apic1(config)# mcp factor 64 apic1(config)# mcp init-delay 180 apic1(config)# mcp transmit-frequency 2 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 319 Configuring Global Policies Configuring the Endpoint Loop Protection Policy Configuring the Endpoint Loop Protection Policy The endpoint loop protection policy is a fabric level policy used in detection of frequent endpoint (host) moves from one fabric port to another. The policy configures what action is to be taken if such an event is detected. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] endpoint loop-detect action {bd-learn-disable | port-disable} Example: apic1(config)# endpoint loop-detect action port-disable Step 3 [no] endpoint loop-detect enable Specifies the action to perform when an endpoint loop is detected. The options are: • bd-learn-disable—Disable MAC address learning on the bridge domain. • port-disable—Disable the port. Allows enabling or disabling of the endpoint loop protection protocol globally for the entire fabric. Example: apic1(config)# endpoint loop-detect enable Step 4 [no] endpoint loop-detect factor number Sets the loop detection multiplication factor. The range is 1 to 255. Example: apic1(config)# endpoint loop-detect factor 64 Step 5 [no] endpoint loop-detect interval seconds Specifies the loop detection interval. The range is 30 to 300 seconds. Example: apic1(config)# endpoint loop-detect interval 60 Examples This example shows how to configure the endpoint loop protection policy. apic1# configure terminal apic1(config)# endpoint loop-detect apic1(config)# endpoint loop-detect apic1(config)# endpoint loop-detect apic1(config)# endpoint loop-detect action port-disable enable factor 64 interval 60 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 320 Configuring Global Policies Configuring IP Aging Configuring IP Aging Overview The IP aging policy tracks and ages unused IPs on an endpoint. Tracking is performed using the endpoint retention policy configured for the BD to send ARP requests (for IPv4) and neighbor solicitations (for IPv6) at 75% of the local endpoint aging interval. When no response is received from an IP, that IP is aged out. This document explains how to configure the IP aging policy. Configuring the IP Aging Policy Using the NX-OS-Style CLI This section explains how to enable and disable the IP aging policy using the CLI. Procedure Step 1 To enable the IP aging policy: Example: ifc1(config)# endpoint ip aging Step 2 To disable the IP aging policy: Example: ifav9-ifc1(config)# no endpoint ip aging Configuring the Dynamic Load Balancer Dynamic load balancing (DLB) adjusts the traffic allocations according to congestion levels. DLB measures the congestion across the available paths and places the flows on the least congested paths, which results in an optimal or near optimal placement of the data. DLB can be configured to place traffic on the available uplinks using the granularity of flows or flowlets. Flowlets are bursts of packets from a flow that are separated by suitably large gaps in time. If the idle interval between two bursts of packets is larger than the maximum difference in latency among available paths, the second burst (or flowlet) can be sent along a different path than the first without reordering packets. This idle interval is measured with a timer called the flowlet timer. Flowlets provide a higher granular alternative to flows for load balancing without causing packet reordering. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 321 Configuring Global Policies Configuring Spanning Tree Protocol Procedure Command or Action Step 1 configure Purpose Enters global configuration mode. Example: apic1# configure Specifies the mode of operation of the load balancer. The modes are: Step 2 [no] system dynamic-load-balance • dynamic-aggressive—The flowlet timeout is a relatively small mode {dynamic-aggressive value. This very fine-grained dynamic load balancing is optimal | dynamic-conservative | for the distribution of traffic, but some packet reordering might link-failure-resiliency | occur. However, the overall benefit to application performance packet-prioritization} is equal to or better than the conservative mode. • dynamic-conservative—The flowlet timeout is a larger value that guarantees packets are not to be re-ordered. The tradeoff is less granular load balancing because new flowlet opportunities are less frequent. • link-failure-resiliency—Static load balancing gives a distribution of flows across the available links that is roughly even. • packet-prioritization—Dynamic Packet Prioritization (DPP) prioritizes short flows higher than long flows; a short flow is less than approximately 15 packets. Because short flows are more sensitive to latency than long ones, DPP can improve overall application performance. apic1(config)# system dynamic-load-balance mode packet-prioritization Examples This example shows how to configure dynamic load balancing with packet prioritization. apic1# configure terminal apic1(config)# system dynamic-load-balance mode packet-prioritization Configuring Spanning Tree Protocol Multiple spanning-tree (MST) enables multiple VLANs to be mapped to the same spanning-tree instance, reducing the number of spanning-tree instances needed to support a large number of VLANs. Note Multiple Spanning Tree (MST) is not supported on interfaces configured with the Per Port VLAN feature (configuring multiple EPGs on a leaf switch using the same VLAN ID with localPort scope). Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 322 Configuring Global Policies Configuring Spanning Tree Protocol Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 spanning-tree mst configuration Enters global configuration mode. Example: apic1(config)# spanning-tree mst configuration Step 3 Enters global configuration mode. [no] bpdu-filter Example: apic1(config-stp)# bpdu-filter Step 4 [no] region region-name Example: apic1(config-stp)# region region1 Step 5 [no] instance instance-id vlan vlan-range Example: For switches to participate in multiple spanning-tree (MST) instances, you must consistently configure the switches with the same MST configuration information. A collection of interconnected switches that have the same MST configuration comprises an MST region. Each region can support up to 65 spanning-tree instances. Maps VLANs to an MST instance. You can assign a VLAN to only one spanning-tree instance at a time. The instance ID range is 1 to 4094. To specify a VLAN range, use a hyphen. apic1(config-stp-region)# instance 2 vlan 1-63 Step 6 revision number Specifies the configuration revision number. The range is 0 to 65535. Example: apic1(config-stp-region)# revision 16 Examples This example shows how to configure an MST spanning-tree policy. apic1# configure terminal apic1(config)# spanning-tree mst configuration apic1(config-stp)# bpdu-filter apic1(config-stp)# region region1 apic1(config-stp-region)# instance 2 vlan 1-63 apic1(config-stp-region)# revision 16 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 323 Configuring Global Policies Configuring IS-IS Configuring IS-IS Intermediate System-to-Intermediate System (IS-IS) is a dynamic link-state routing protocol that can detect changes in the network topology and calculate loop-free routes to other nodes in the network. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 template isis-fabric isis-fabric-template-name Enters Intermediate System-to-Intermediate System (IS-IS) configuration mode and creates an IS-IS fabric template (policy). Example: apic1(config)# template isis-fabric polIsIs Step 3 [no] lsp-fast-flood Example: apic1(config-template-isis-fabric)# lsp-fast-flood Step 4 Enables the fast-flood feature, which improves convergence time when new link-state packets (LSPs) are generated in the network and shortest path first (SPF) is triggered by the new LSPs. We recommend that you enable the fast-flooding of LSPs before the router runs the SPF computation, to ensure that the whole network achieves a faster convergence time. [no] lsp-gen-interval level-1 lsp-max-wait Configures the IS-IS throttle for LSP generation. The parameters are as follows: [lsp-initial-wait lsp-second-wait] Example: apic1(config-template-isis-fabric)# lsp-gen-interval level-1 500 500 500 • lsp-max-wait—The maximum wait between the trigger and LSP generation. • lsp-initial-wait—The initial wait between the trigger and LSP generation. • lsp-second-wait—The second wait used for LSP throttle during backoff. The lsp-max-wait parameter is required. The other two parameters are optional but must appear together. The range for each is 50 to 120000 milliseconds. Step 5 [no] lsp-mtu mtu Sets the maximum transmission unit (MTU) size of IS-IS hello packets. The range is 256 to 4352. Example: IS-IS hello packets are used to discover and maintain adjacencies. By default, the hello packets are padded to the full maximum transmission unit (MTU) size to allow for early detection of errors due to transmission apic1(config-template-isis-fabric)# lsp-mtu 2048 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 324 Configuring Global Policies Configuring IS-IS Command or Action Purpose problems with large frames or due to mismatched MTUs on adjacent interfaces. However, IS-IS adjacency formation may fail due to MTU mismatch on a link, requiring the adjustment of the MTU size. Step 6 [no] spf-interval level-1 spf-max-wait [spf-initial-wait spf-second-wait] Example: apic1(config-template-isis-fabric)# spf-interval level-1 500 500 500 Configures the interval between LSA arrivals. The parameters are as follows: • spf-max-wait—The maximum wait between the trigger and SPF computation. • spf-initial-wait—The initial wait between the trigger and SPF computation. • spf-second-wait—The second wait used for SPF computation during backoff. The spf-max-wait parameter is required. The other two parameters are optional but must appear together. The range for each is 50 to 120000 milliseconds. Step 7 Returns to global configuration mode. exit Example: apic1(config-template-isis-fabric)# exit Step 8 Creates a pod group template (policy). template pod-group pod-group-template-name Example: apic1(config)# template pod-group allPods Step 9 inherit pod-group pod-group-name Configures the template pod-group to use the previously configured isis-fabric template (policy). Example: apic1(config-pod-group)# inherit isis-fabric polIsIs Step 10 Returns to global configuration mode. exit Example: apic1(config-pod-group)# exit Step 11 pod-profile pod-profile-name Configures a pod profile. Example: apic1(config)# pod-profile all Step 12 pods {pod-range-1-255 | all} Configures a set of pods. Example: apic1(config-pod-profile)# pods all Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 325 Configuring Global Policies Configuring BGP Route Reflectors Step 13 Command or Action Purpose inherit pod-group pod-group-name Configures the pod-profile to use the previously configured pod group. Example: apic1(config-pod-profile-pods)# inherit pod-group allPods Step 14 Returns to EXEC mode. end Example: apic1(config-pod-profile-pods)# end Examples This example shows how to configure IS-IS. aapic1# configure apic1(config)# template isis-fabric polIsIs apic1(config-template-isis-fabric)# lsp-fast-flood apic1(config-template-isis-fabric)# lsp-gen-interval level-1 500 500 500 apic1(config-template-isis-fabric)# lsp-mtu 2048 apic1(config-template-isis-fabric)# spf-interval level-1 500 500 500 apic1(config-template-isis-fabric)# exit apic1(config)# template pod-group allPods apic1(config-pod-group)# inherit isis-fabric polIsIs apic1(config-pod-group)# exit apic1(config)# pod-profile all apic1(config-pod-profile)# pods all apic1(config-pod-profile-pods)# inherit pod-group allPods apic1(config-pod-profile-pods)# end apic1# Configuring BGP Route Reflectors The ACI fabric route reflectors use multiprotocol Border Gateway Protocol (MP-BGP) to distribute external routes within the fabric. To enable route reflectors in the ACI fabric, the fabric administrator must select the spine switches that will be the route reflectors, and provide the autonomous system (AS) number. For redundancy purposes, more than one spine is configured as a router reflector node (one primary and one secondary reflector). Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 326 Configuring Global Policies Decommissioning a Node Step 2 Command or Action Purpose bgp-fabric Enters BGP configuration mode for the fabric. Example: apic1(config)# bgp-fabric Step 3 asn asn-value Configures the BGP Autonomous System number (ASN), which uniquely identifies an autonomous system. The ASN is between 1 and 4294967295. Example: Step 4 apic1(config-bgp-fabric)# asn 123456789 We recommend that you enable the fast-flooding of LSPs before the router runs the SPF computation, to ensure that the whole network achieves a faster convergence time. [no] route-reflector spine list Configure up to two spine nodes as route reflectors. For redundancy ,you should configure primary and secondary route reflectors. Example: apic1(config-bgp-fabric)# route-reflector spine spine1,spine2 Examples This example shows how to configure spine1 and spine2 as BGP route reflectors. apic1# configure apic1(config)# bgp-fabric apic1(config-bgp-fabric)# asn 123456789 apic1(config-bgp-fabric)# route-reflector spine spine1,spine2 apic1(config-bgp-fabric)# exit apic1(config)# Decommissioning a Node Two levels of decommissioning are supported: • Regular—Similar to disabling the node. After being decommissioned, the node cannot rejoin the fabric until the no decommission command is executed. • Complete—When the node is decommissioned, all fabric configuration related to the node is cleared. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] decommission {controller | switch} node-id [remove-from-controller] Decommissions the specified node. Note that controller node ID numbers are between 1 and Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 327 Configuring Global Policies Configuring Power Management Command or Action Purpose Example: 100, while switch node ID numbers are between 101 and 4000. apic1(config)# decommission switch 104 remove-from-controller Examples This example shows how to perform a complete decommissioning of node 104 (a switch) and recommission node 5 (a controller), which was decommissioned with the regular level. apic1# configure apic1(config)# decommission switch 104 remove-from-controller apic1(config)# no decommission controller 5 Configuring Power Management Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] power redundancy-policy policy-name Creates or configures a power supply redundancy policy. Example: apic1(config)# power redundancy-policy myPowerPolicy Step 3 [no] description text Adds a description for this power supply redundancy policy. If the text includes spaces, it must be enclosed in single quotes. Example: apic1(config-power)# description 'This is my power redundancy policy' Step 4 [no] redundancy-mode {combined | ps-redundant | redundant} Specifies power supply redundancy mode. Example: apic1(config-power)# redundancy-mode ps-redundant Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 328 • combined— This mode does not provide power redundancy. The available power is the total power capacity of all power supplies. • ps-redundant—This mode provides an extra power supply in case an active power supply goes down. The power supply that can supply the most power operates in Configuring Global Policies Configuring a Scheduler Command or Action Purpose standby mode. The other one or two power supplies are active. The available power is the amount of power provided by the active power supply units. • redundant—This mode combines power supply redundancy and input source redundancy, which means that the chassis has an extra power supply and each half of each power supply is connected to one electrical grid while the other half of each power supply is connected to the other electrical grid. The available power is the lesser of the available power for power supply mode and input source mode. Examples This example shows how to configure a power supply redundancy policy for the ps-redundant mode. apic1# configure apic1(config)# power redundancy-policy myPowerPolicy apic1(config-pod)# isis fabric apic1(config-power)# description 'This is my power redundancy policy' apic1(config-power)# redundancy-mode ps-redundant Configuring a Scheduler A schedule allows operations, such as configuration import/export or tech support collection, to occur during one or more specified windows of time. A schedule contains a set of time windows (occurrences). These windows can be one time only or can recur at a specified time and day each week. The options defined in the window, such as the duration or the maximum number of tasks to be run, determine when a scheduled task will execute. For example, if a change cannot be deployed during a given maintenance window because the maximum duration or number of tasks has been reached, that deployment is carried over to the next maintenance window. Each schedule checks periodically to see whether the APIC has entered one or more maintenance windows. If it has, the schedule executes the deployments that are eligible according to the constraints specified in the maintenance policy. A schedule contains one or more occurrences, which determine the maintenance windows associated with that schedule. An occurrence can be one of the following: • Absolute (One Time) Window—An absolute window defines a schedule that will occur only once. This window continues until the maximum duration of the window or the maximum number of tasks that can be run in the window has been reached. • Recurring Window—A recurring window defines a repeating schedule. This window continues until the maximum number of tasks or the end of the day specified in the window has been reached. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 329 Configuring Global Policies Configuring a Scheduler Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] scheduler schedule-name Creates a new scheduler or configures an existing scheduler. Example: apic1(config)# scheduler controller schedule myScheduler Step 3 [no] description text Example: Adds a description for this scheduler. If the text includes spaces, it must be enclosed in single quotes. apic1(config-scheduler)# description 'This is my scheduler' Step 4 [no] absolute window window-name Creates an absolute (one time) window schedule. Example: apic1(config-scheduler)# absolute window myAbsoluteWindow Step 5 [no] max concurrent nodes count Example: Sets the maximum number of nodes (tasks) that can be processed concurrently. The range is 0 to 65535. Set to 0 for unlimited nodes. apic1(config-scheduler-absolute)# max concurrent nodes 300 Step 6 [no] max running time time Example: Sets the maximum running time for tasks in the format dd:hh:mm:ss. The range is 0 to 65535. Set to 0 for no time limit. apic1(config-scheduler-absolute)# max running time 00:01:30:00 Step 7 [no] time start time Sets the starting time in the format [[[yyyy:]mmm:]dd:]HH:MM. Example: apic1(config-scheduler-absolute)# time start 2016:jan:01:12:01 Step 8 exit Returns to scheduler configuration mode. Example: apic1(config-scheduler-absolute)# exit Step 9 [no] recurring window window-name Example: apic1(config-scheduler)# recurring window myRecurringWindow Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 330 Creates a recurring window schedule. Configuring Global Policies Configuring a Scheduler Step 10 Command or Action Purpose [no] max concurrent nodes count Sets the maximum number of nodes (tasks) that can be processed concurrently. The range is 0 to 65535. Set to 0 for unlimited nodes. Example: apic1(config-scheduler-recurring)# max concurrent nodes 300 Step 11 [no] max running time time Example: Sets the maximum running time for tasks in the format dd:hh:mm:ss. The range is 0 to 65535. Set to 0 for no time limit. apic1(config-scheduler-recurring)# max running time 00:01:30:00 Step 12 [no] time start {daily HH:MM | weekly (See Sets the period (daily or weekly) and starting time. If weekly is selected, choose from these usage) HH:MM} options: Example: apic1(config-scheduler-recurring)# time start weekly wednesday 12:30 • monday • tuesday • wednesday • thursday • friday • saturday • sunday • even-day • odd-day • every-day Examples This example shows how to configure a recurring scheduler to run every Wednesday. apic1# configure apic1(config)# scheduler controller schedule myScheduler apic1(config-scheduler)# description 'This is my scheduler' apic1(config-scheduler)# recurring window myRecurringWindow apic1(config-scheduler-recurring)# max concurrent nodes 300 apic1(config-scheduler-recurring)# max running time 00:01:30:00 apic1(config-scheduler-recurring)# time start weekly wednesday 12:30 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 331 Configuring Global Policies Configuring System MTU Configuring System MTU Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] system jumbomtu size Example: Sets the maximum transmit unit (MTU) for host facing ports. The range is 576 to 9000 bytes; the default is 9000. apic1(config)# system jumbomtu 9000 Examples This example shows how to configure the system MTU size. apic1# configure terminal apic1(config)# system jumbomtu 9000 About PTP Precision Time Protocol (PTP) is a time synchronization protocol defined in IEEE 1588 for nodes distributed across a network. With PTP, it is possible to synchronize distributed clocks with an accuracy of less than 1 microsecond via Ethernet networks. PTP’s accuracy comes from the hardware support for PTP in the ACI fabric spines and leafs. It allows the protocol to accurately compensate for message delays and variation across the network. PTP is a distributed protocol that specifies how real-time PTP clocks in the system synchronize with each other. These clocks are organized into a master-slave synchronization hierarchy with the grandmaster clock, which is the clock at the top of the hierarchy, determining the reference time for the entire system. Synchronization is achieved by exchanging PTP timing messages, with the members using the timing information to adjust their clocks to the time of their master in the hierarchy. PTP operates within a logical scope called a PTP domain. The PTP process consists of two phases: establishing the master-slave hierarchy and synchronizing the clocks. Within a PTP domain, each port of an ordinary or boundary clock follows this process to determine its state: • Examines the contents of all received announce messages (issued by ports in the master state). • Compares the data sets of the foreign master (in the announce message) and the local clock for priority, clock class, accuracy, and so on. • Determines its own state as either master or slave. After the master-slave hierarchy has been established, the clocks are synchronized as follows: Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 332 Configuring Global Policies About PTP • The master sends a synchronization message to the slave and notes the time it was sent. • The slave receives the synchronization message and notes the time that it was received. For every synchronization message, there is a follow-up message. Hence, the number of sync messages should be equal to the number of follow-up messages. • The slave sends a delay-request message to the master and notes the time it was sent. • The master receives the delay-request message and notes the time it was received. • The master sends a delay-response message to the slave. The number of delay request messages should be equal to the number of delay response messages. • The slave uses these timestamps to adjust its clock to the time of its master. In ACI fabric, when PTP feature is globally enabled in APIC, the software automatically enables PTP on specific interfaces of all the supported spines and leafs. This auto-configuration ensures that PTP is optimally enabled on all the supported nodes. In the absence of an external grandmaster clock, one of the spine switch is chosen as the grandmaster. The master spine is given a different PTP priority as compared to the other spines and leaf switches so that they will act as PTP slaves. This way we ensure that all the leaf switches in the fabric synchronize to the PTP clock of the master spine. If an external Grandmaster clock is connected to the spines, the spine syncs to the external GM and in turn acts as a master to the leaf nodes. PTP Default Settings The following table lists the default settings for PTP parameters. Parameters Default PTP device type Boundary clock PTP clock type Two-step clock PTP domain 0 PTP priority 1 value when advertising the clock 255 PTP priority 2 value when advertising the clock 255 PTP announce interval 1 log second PTP announce timeout 3 announce intervals PTP delay-request interval 0 log seconds PTP sync interval -2 log seconds PTP VLAN 1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 333 Configuring Global Policies Guidelines and Limitations Note PTP operates only in boundary clock mode. Cisco recommends deployment of a Grand Master Clock (10 MHz) upstream, with servers containing clocks requiring synchronization connected to the switch. PTP Verification Command Purpose show ptp brief Displays the PTP status. show ptp clock Displays the properties of the local clock, including clock identity. show ptp clock foreign-masters record interface ethernet slot/port Displays the state of foreign masters known to the PTP process. For each foreign master, the output displays the clock identity, basic clock properties, and whether the clock is being used as a grandmaster. show ptp corrections Displays the last few PTP corrections. show ptp counters [all |interface Ethernet slot/port] Displays the PTP packet counters for all interfaces or for a specified interface. show ptp parent Displays the properties of the PTP parent. Guidelines and Limitations Follow these guidelines and limitations: • Latency requires all the nodes in the fabric to be synchronized using Precision Time Protocol (PTP). • Beginning with Cisco ACI release 3.0(1x), latency measurement and PTP are only supported on the following: ◦N9K-X9732C-EX, ◦N9K-X9736C-EX ◦N9K-C93180YC-FX ◦N9K-C93108TC-FX ◦N9K-C93108TC-EX ◦N9K-C93180YC-EX ◦N9K-X9736C-FX • Latency measurement is supported only for the packets that ingress, egress and transit via EX or FX based TORs. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 334 Configuring Global Policies Configuring PTP Using the NX-OS CLI • All the spine nodes in the fabric should have EX or FX based line cards to support PTP. • PTP and the latency feature is not supported on any NS ToRs or Spines. In the presence of non-EX/FX TORs in the fabric, it is recommended to have the external GM connectivity to all the spines to ensure that the PTP time is synced across all the supported TORs. • External Grandmaster (GM) clock is not mandatory for PTP in a single Pod. If there is no external GM connected to the ACI fabric, one of the Spine nodes acts as the GM. This Spine has a PTP priority1 value as 254. All the other Spines and Leaf switches in the fabric will synchronize their clock to this Master Spine clock. If the external GM is connected later to the Spine, it should have a priority value less than 254 for it to act as the GM for the entire fabric. • External Grandmaster clock is mandatory for PTP in a Multi-Pod scenario. In addition, External GM needs to be connected to the IPN such that the Grandmaster clock is the master to the spines in different PODs. The Spines connected to IPN will act as the boundary clock and all the nodes within the POD will sync their clock this spine. • PTP operates only in boundary clock mode. End-to-end transparent clock and peer-to-peer transparent clock modes are not supported. • PTP supports transport over User Datagram Protocol (UDP). Transport over Ethernet is not supported. • ACI PTP supports multicast communication only, unicast mode is not supported. Configuring PTP Using the NX-OS CLI Procedure Step 1 Enable PTP. Example: Enable ptp: ======== apic# configure terminal apic(config)# ptp Disable ptp: ======== apic# configure terminal apic(config)# no ptp Step 2 To verify PTP on ACI switches: Example: leaf1# show ptp brief PTP port status ----------------------Port State ------- -------------Eth1/49 Slave leaf1# leaf1# leaf1# show ptp clock PTP Device Type: Boundary clock Clock Identity : 0c:75:bd:ff:fe:03:1d:10 Clock Domain: 0 Number of PTP ports: 1 Priority1 : 255 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 335 Configuring Global Policies Configuring PTP Using the NX-OS CLI Priority2 : 255 Clock Quality: Class : 248 Accuracy : 254 Offset (log variance) : 65535 Offset From Master : 32 Mean Path Delay : 128 Steps removed : 1 Local clock time:Thu Jul 27 19:43:42 2017 leaf1# leaf1# show ptp clock foreign-masters record interface ethernet 1/49 P1=Priority1, P2=Priority2, C=Class, A=Accuracy, OSLV=Offset-Scaled-Log-Variance, SR=Steps-Removed GM=Is grandmaster --------Interface --------Eth1/49 ----------------------Clock-ID ----------------------d4:6d:50:ff:fe:e6:4d:3f --P1 --- ---P2 ---- ---C ---- --A --- ----OSLV ----- -------SR -------- 254 255 248 254 65535 0 GM leaf1# leaf1# leaf1# show ptp corrections PTP past corrections ----------------------------------------------------------------------------------Slave Port SUP Time Correction(ns) MeanPath Delay(ns) ---------- ------------------------------- ------------------ -----------------Eth1/49 Thu Jul 27 19:44:11 2017 364281 36 152 Eth1/49 Thu Jul 27 19:44:11 2017 114565 16 132 Eth1/49 Thu Jul 27 19:44:10 2017 862912 8 132 Eth1/49 Thu Jul 27 19:44:10 2017 610823 8 132 Eth1/49 Thu Jul 27 19:44:10 2017 359557 16 132 Eth1/49 Thu Jul 27 19:44:10 2017 109937 8 132 Eth1/49 Thu Jul 27 19:44:09 2017 858113 16 132 Eth1/49 Thu Jul 27 19:44:09 2017 606536 16 132 Eth1/49 Thu Jul 27 19:44:09 2017 354837 -16 132 Eth1/49 Thu Jul 27 19:44:09 2017 104226 24 148 Eth1/49 Thu Jul 27 19:44:08 2017 853263 24 148 Eth1/49 Thu Jul 27 19:44:08 2017 601780 16 148 Eth1/49 Thu Jul 27 19:44:08 2017 349639 -4 148 Eth1/49 Thu Jul 27 19:44:08 2017 99970 16 144 Eth1/49 Thu Jul 27 19:44:07 2017 848507 0 144 Eth1/49 Thu Jul 27 19:44:07 2017 596143 24 144 Eth1/49 Thu Jul 27 19:44:07 2017 344808 4 144 Eth1/49 Thu Jul 27 19:44:07 2017 93156 -16 140 Eth1/49 Thu Jul 27 19:44:06 2017 843263 24 140 Eth1/49 Thu Jul 27 19:44:06 2017 590189 8 140 leaf1# leaf1# leaf1# show ptp counters all PTP Packet Counters of Interface Eth1/49: ---------------------------------------------------------------Packet Type TX RX -----------------------------------------------------Announce 56 5424 Sync 441 43322 FollowUp 441 43321 Delay Request 7002 0 Delay Response 0 7002 PDelay Request 0 0 PDelay Response 0 0 PDelay Followup 0 0 Management 0 0 ---------------------------------------------------------------leaf1# leaf1# Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 336 Configuring Global Policies Configuring PTP Using the NX-OS CLI leaf1# show ptp parent PTP PARENT PROPERTIES Parent Clock: Parent Clock Identity: d4:6d:50:ff:fe:e6:4d:3f Parent Port Number: 258 Observed Parent Offset (log variance): N/A Observed Parent Clock Phase Change Rate: N/A Grandmaster Clock: Grandmaster Clock Identity: d4:6d:50:ff:fe:e6:4d:3f Grandmaster Clock Quality: Class: 248 Accuracy: 254 Offset (log variance): 65535 Priority1: 254 Priority2: 255 leaf1# Step 3 To verify troubleshooting steps: Example: apic1# show troubleshoot eptoep session eptoep latency Source --> Destination Last Collection(30 seconds) +--------------------+-------------------------------+--------------+ | Average (microsec) | Standard Deviation (microsec) | Packet Count | +--------------------+-------------------------------+--------------+ | 18 | 24 | 1086 | | | | | +--------------------+-------------------------------+--------------+ Cumulative +--------------------+----------------+--------------------+ | Average (microsec) | Max (microsec) | Total Packet Count | +--------------------+----------------+--------------------+ | 18 | 202 | 6117438 | | | | | +--------------------+----------------+--------------------+ Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 337 Configuring Global Policies Configuring PTP Using the NX-OS CLI Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 338 CHAPTER 12 Configuring Cisco Tetration Analytics • Overview, page 339 • Configuring Cisco Tetration Analytics Using the NX-OS Style CLI, page 339 Overview This article provides examples of how to configure Cisco Tetration Analytics when using the Cisco APIC. The following information applies when configuring Cisco Tetration Analytics. • An inband management IP address must be configured on each leaf where the Cisco Tetration Analytics agent is active. • Define an analytics policy and specify the destination IP address of the Cisco Tetration Analytics server. • Create a switch profile and include the policy group created in the previous step. Configuring Cisco Tetration Analytics Using the NX-OS Style CLI Procedure Step 1 Command or Action Purpose configure terminal Enters global configuration mode. Example: apic1# configure terminal Step 2 analytics cluster cluster_name Create the analytics policy. Example: apic1(config)# analytics cluster cluster1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 339 Configuring Cisco Tetration Analytics Configuring Cisco Tetration Analytics Using the NX-OS Style CLI Step 3 Command or Action Purpose flow-exporter server_name Configure external analytics information. Example: apic1(config-analytics)# flow-exporter server1 Step 4 destination ip_address Configure the destination port. Example: apic1(config-analytics-cluster-exporter)# destination 192.0.2.1 Step 5 Exit command mode. exit Example: # apic1(config-analytics-cluster-exporter)# exit Step 6 Exit command mode. exit Example: apic1(config-analytics)# Step 7 exit fabric-internal Enters fabric internal configuration mode. Example: apic1(config)# fabric-internal Step 8 template leaf-policy-group leaf_group_name Define leaf policy group. Example: apic1(config-fabric-internal)# template leaf-policy-group lpg1 Step 9 inherit analytics-policy cluster cluster_name server server_name Associate analytics policy to leaf policy group. Example: apic1(config-leaf-policy-group)# inherit analytics-policy cluster cluster1 server server1 Step 10 exit Exit command mode. Example: apic1(config-leaf-policy-group)# exit Step 11 leaf-profile lleaf_profile_name Define leaf profile. Example: apic1(config-fabric-internal)# leaf-profile lp1 Step 12 leaf-group leaf_group_name Example: apic1(config-leaf-profile)# leaf-group lg1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 340 Define leaf group. Configuring Cisco Tetration Analytics Configuring Cisco Tetration Analytics Using the NX-OS Style CLI Step 13 Command or Action Purpose leaf-policy-group leaf_policy_group_name Associate leaf policy group to leaf group. Example: apic1(config-leaf-group)# leaf-policy-group lpg1 Step 14 leaf leaf_group_number Add nodes to leaf group. Example: apic1(config-leaf-group)# leaf 101 Step 15 Display analytics. show analytics Example: apic1# show analytics Cluster : Config Server Name : Destination IP : Destination Port : DSCP : 33 33 192.0.2.1 10000 VA Cluster : server333 Config Server Name : server333 Destination IP : 192.0.2.2 Destination Port : 5640 DSCP : AF21 Cluster : tet3 Config Server Name : server2 Destination IP : 192.0.2.3 Destination Port : 5640 DSCP : AF12 Cluster : tet2 Config Server Name : server1 Destination IP : 192.0.2.4 Destination Port : 5640 DSCP : AF11 Step 16 show running-config analytics Display running configuration analytics. Example: apic1# show running-config analytics # Command: show running-config analytics # Time: Wed May 25 21:14:43 2016 analytics cluster 33 flow-exporter 33 destination 192.0.2.1 destination-port 10000 dscp VA exit exit analytics cluster server333 flow-exporter server333 destination 192.0.2.2 destination-port 5640 dscp AF21 exit exit analytics cluster tet3 flow-exporter server2 destination 192.0.2.3 destination-port 5640 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 341 Configuring Cisco Tetration Analytics Configuring Cisco Tetration Analytics Using the NX-OS Style CLI Command or Action dscp AF12 exit exit analytics cluster tet2 flow-exporter server1 destination 192.0.2.4 destination-port 5640 dscp AF11 exit exit apic1# Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 342 Purpose CHAPTER 13 Configuring NetFlow • About NetFlow, page 343 • Configuring a NetFlow Exporter Policy for Virtual Machine Networking Using the NX-OS-Style CLI, page 344 • Configuring the NetFlow and Tetration Analytics Feature Priority Through a Node Control Policy Using the NX-OS-Style CLI, page 344 • Configuring a NetFlow Node Policy Using the NX-OS-Style CLI, page 345 • Configuring NetFlow Infra Selectors Using the NX-OS-Style CLI, page 346 • Configuring NetFlow Overrides Using the NX-OS-Style CLI, page 348 • Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style CLI, page 348 • Consuming a NetFlow Exporter Policy Under a VMM Domain Using the NX-OS-Style CLI for VMware VDS, page 351 • Enabling or Disabling NetFlow on an Endpoint Group Using the NX-OS-Style CLI for VMware VDS, page 352 About NetFlow The NetFlow technology provides the metering base for a key set of applications, including network traffic accounting, usage-based network billing, network planning, as well as denial of services monitoring, network monitoring, outbound marketing, and data mining for both service providers and enterprise customers. Cisco provides a set of NetFlow applications to collect NetFlow export data, perform data volume reduction, perform post-processing, and provide end-user applications with easy access to NetFlow data. If you have enabled NetFlow monitoring of the traffic flowing through your datacenters, this feature enables you to perform the same level of monitoring of the traffic flowing through the Cisco Application Centric Infrastructure (Cisco ACI) fabric. Instead of hardware directly exporting the records to a collector, the records are processed in the supervisor engine and are exported to standard NetFlow collectors in the required format. For information about configuring NetFlow with virtual machine networking, see the Cisco ACI Virtualization Guide. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 343 Configuring NetFlow Configuring a NetFlow Exporter Policy for Virtual Machine Networking Using the NX-OS-Style CLI Note NetFlow is only supported on EX switches. See the Cisco NX-OS Release Notes for Cisco Nexus 9000 Series ACI-Mode Switches document for the release that you have installed for a list of the supported EX switches. Configuring a NetFlow Exporter Policy for Virtual Machine Networking Using the NX-OS-Style CLI The following example procedure uses the NX-OS-style CLI to configure a NetFlow exporter policy for virtual machine networking. Procedure Step 1 Enter the configuration mode. Example: apic1# config Step 2 Configure the exporter policy. Example: apic1(config)# flow vm-exporter vmExporter1 destination address 2.2.2.2 transport udp 1234 apic1(config-flow-vm-exporter)# source address 4.4.4.4 apic1(config-flow-vm-exporter)# exit apic1(config)# exit Configuring the NetFlow and Tetration Analytics Feature Priority Through a Node Control Policy Using the NX-OS-Style CLI The following example procedure uses the NX-OS-style CLI to configure the NetFlow and Tetration Analytics feature priority through a node control policy: Procedure Step 1 Enter the configuration mode. Example: apic1# config Step 2 Create a node control policy. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 344 Configuring NetFlow Configuring a NetFlow Node Policy Using the NX-OS-Style CLI Example: apic1(config)# node-control policy pol1 Step 3 Set NetFlow as the priority feature. Example: apic1(config-node)# feature netflow Step 4 Exit the node control policy configuration. Example: apic1(config-node)# end Step 5 Deploy the policy to node 101 and node 102. Example: ifav-isim15-ifc1(config)# fabric-internal ifav-isim15-ifc1(config-fabric-internal)# template leaf-policy-group lpg1 ifav-isim15-ifc1(config-leaf-policy-group)# inherit node-control-policy pol1 ifav-isim15-ifc1(config-leaf-policy-group)# exit ifav-isim15-ifc1(config-fabric-internal)# leaf-profile leafProfile1 ifav-isim15-ifc1(config-leaf-profile)# leaf-group leafgrp1 ifav-isim15-ifc1(config-leaf-group)# leaf 101 ifav-isim15-ifc1(config-leaf-group)# leaf 102 ifav-isim15-ifc1(config-leaf-group)# leaf-policy-group lpg1 ifav-isim15-ifc1(config-leaf-group)# end Configuring a NetFlow Node Policy Using the NX-OS-Style CLI The following example procedure uses the NX-OS-style CLI to configure a NetFlow node policy: Procedure Step 1 Enter the configuration mode. Example: apic1# config Step 2 Configure the node policy. Example: apic1(config)# flow node-policy nodePol apic1(config-flow-node-pol)# flow timeout collection 100 apic1(config-flow-node-pol)# flow timeout template 123 apic1(config-flow-node-pol)# exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 345 Configuring NetFlow Configuring NetFlow Infra Selectors Using the NX-OS-Style CLI Configuring NetFlow Infra Selectors Using the NX-OS-Style CLI You can use the NX-OS-style CLI to configure NetFlow infra selectors. The infra selectors are used for attaching a Netflow monitor to a PHY, port channel, virtual port channel, fabric extender (FEX), or port channel fabric extender (FEXPC) interface. The following example CLI commands show how to configure NetFlow infra selectors using the NX-OS-style CLI: Procedure Step 1 Enter the configuration mode. Example: apic1# config Step 2 Create a NetFlow exporter policy. Example: In the following commands, the destination endpoint group is the endpoint group that the exporter sits behind. This endpoint group can also be an external Layer 3 endpoint group. apic1(config)# flow exporter apic1(config-flow-exporter)# apic1(config-flow-exporter)# apic1(config-flow-exporter)# apic1(config-flow-exporter)# apic1(config-flow-exporter)# Step 3 infraExporter1 destination address 1.2.3.4 transpo udp 1234 destination epg tenant tn2 application ap2 epg epg2 vrf member tenant tn2 vrf vrf2 version v9 source address 1.1.1.1 exit Create a second NetFlow exporter policy. Example: In the following commands, the destination endpoint group is the endpoint group that the exporter sits behind, which in this case is an external Layer 3 endpoint group. apic1(config)# flow exporter infraExporter2 apic1(config-flow-exporter)# transport udp 9990 apic1(config-flow-exporter)# destination address 2001:db5:a0c:1f0::2 apic1(config-flow-exporter)# destination external-l3 epg tenant tn2 vrf v2 epg accounting-inst apic1(config-flow-exporter)# vrf member tenant tn2 vrf vrf2 apic1(config-flow-exporter)# version v5 apic1(config-flow-exporter)# source address 2001:db8:a0b:12f0::1 apic1(config-flow-exporter)# exit Step 4 Create a NetFlow record policy. Example: apic1(config)# flow record apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# infraRecord1 match dst-ip match dst-ipv4 match dst-ipv6 match dst-mac match dst-port match ethertype match proto match src-ip match src-ipv4 match src-ipv6 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 346 Configuring NetFlow Configuring NetFlow Infra Selectors Using the NX-OS-Style CLI apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# Step 5 match src-mac match src-port match tos match vlan collect count-bytes collect count-pkts collect pkt-disp collect sampler-id collect src-intf collect tcp-flags collect ts-first collect ts-recent exit Create a NetFlow monitor policy. Example: apic1(config)# flow monitor apic1(config-flow-monitor)# apic1(config-flow-monitor)# apic1(config-flow-monitor)# apic1(config-flow-monitor)# infraMonitor1 record infraRecord1 exporter infraExporter1 exporter infraExporter2 exit You can attach a maximum of two exporters. Step 6 Create an interface policy group (AccPortGrp). Example: apic1(config)# template policy-group pg1 apic1(config-pol-grp-if)# ip flow monitor infraMonitor1 apic1(config-pol-grp-if)# ipv6 flow monitor infraMonitor2 apic1(config-pol-grp-if)# exit You can have one monitor policy per address family (IPv4 and IPv6). Step 7 Create a node profile and infra selectors. Example: apic1(config)# leaf-profile lp1 apic1(config-leaf-profile)# leaf-group lg1 apic1(config-leaf-group)# leaf 101 apic1(config-leaf-profile)# exit apic1(config)# leaf-interface-profile lip1 apic1(config-leaf-if-profile)# exit apic1(config)# leaf-interface-profile lip1 apic1(config-leaf-if-profile)# leaf-interface-group lig1 apic1(config-leaf-if-group)# interface ethernet 1/5 apic1(config-leaf-if-profile)# policy-group pg1 apic1(config-leaf-if-profile)# exit apic1(config-leaf-profile)# exit Step 8 Create a port channel policy group (AccBndlGrp). Example: apic1(config)# template port-channel po6 apic1(config-if)# ip flow monitor infraMonitor1 apic1(config-if)# ipv6 flow monitor infraMonitor1 apic1(config-if)# exit apic1(config-leaf-profile)# leaf-profile lp2 apic1(config-leaf-group)# leaf-group lg2 apic1(config-leaf-profile)# leaf 101 apic1(config-leaf-profile)# exit apic1(config)# leaf-interface-profile lip2 apic1(config-leaf-if-profile)# exit apic1(config)# leaf-interface-profile lip2 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 347 Configuring NetFlow Configuring NetFlow Overrides Using the NX-OS-Style CLI apic1(config-leaf-if-profile)# leaf-interface-group lig2 apic1(config-leaf-if-group)# interface ethernet 1/6 apic1(config-leaf-if-profile)# channel-group po6 apic1(config-leaf-if-profile)# exit You can have one monitor policy per address family (IPv4 and IPv6). The interfaces can also be vPCs. Configuring NetFlow Overrides Using the NX-OS-Style CLI The following procudure configures NetFlow overrides using the NX-OS-Style CLI: Procedure Step 1 Enter the configuration mode. Example: apic1# config Step 2 Create the override. Example: apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant tn2 vrf vrf2 apic1(config-leaf)# exit apic1(config)# interface ethernet 1/15 apic1(config-if)# ip flow monitor infraMonitor1 apic1(config-if)# ipv6 flow monitor infraMonitor2 apic1(config-if)# exit apic1(config)# exit apic1# exit You can have one monitor policy per address family (IPv4 and IPv6). The interfaces can also be vPCs. Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style CLI The following example procedure uses the NX-OS-style CLI to configure the NetFlow tenant hierarchy: Procedure Step 1 Enter the configuration mode. Example: apic1# config Step 2 Create a tenant and bridge domain, and add them to a VRF. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 348 Configuring NetFlow Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style CLI Example: apic1(config)# tenant tn2 apic1(config-tenant)# vrf context vrf2 apic1(config-tenant-vrf)# exit apic1(config-tenant)# bridge-domain bd2 apic1(config-tenant-bridge-domain)# vrf member vrf2 apic1(config-tenant-bridge-domain)# exit apic1(config-tenant)# bridge-domain bd3 apic1(config-tenant-bridge-domain)# vrf member vrf2 apic1(config-tenant-bridge-domain)# exit Step 3 Create an application endpoint group behind which the exporter resides. Example: apic1(config-tenant)# application ap2 apic1(config-tenant-app)# epg epg2 apic1(config-tenant-app)# bridge-domain member bd2 apic1(config-tenant-app-bridge-domain)# exit apic1(config-tenant-app)# exit Step 4 Create a second application endpoint group behind which the exporter resides. Example: apic1(config-tenant)# application ap3 apic1(config-tenant-app)# epg epg3 apic1(config-tenant-app)# bridge-domain member bd3 apic1(config-tenant-app-bridge-domain)# exit apic1(config-tenant-app)# exit Step 5 Attach a NetFlow monitor policy on the bridge domains. Example: apic1(config)# interface bridge-domain bd2 apic1(config-if)# ipv6 flow monitor tnMonitor1 apic1(config-if)# ip flow monitor tnMonitor1 apic1(config-if)# layer2-switched flow monitor tnMonitor1 apic1(config-if)# exit apic1(config)# interface bridge-domain bd3 apic1(config-if)# ipv6 flow monitor tnMonitor1 apic1(config-if)# ip flow monitor tnMonitor1 apic1(config-if)# exit You can have one monitor policy per address family (IPv4 and IPv6). The interfaces can also be vPCs. Step 6 Create the Netflow exporter policy. Example: In the following commands, the destination endpoint group is the endpoint group that the exporter sits behind. This endpoint group can also be an external Layer 3 endpoint group. apic1(config)# flow exporter apic1(config-flow-exporter)# apic1(config-flow-exporter)# apic1(config-flow-exporter)# apic1(config-flow-exporter)# apic1(config-flow-exporter)# apic1(config-flow-exporter)# apic1(config-flow-exporter)# Step 7 tnExporter1 transport udp 1234 destination address 2.2.2.2 destination epg tenant tn2 application ap2 epg epg2 vrf member tenant tn2 vrf vrf2 version v9 source address 1.1.1.1 exit Create a second Netflow exporter policy. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 349 Configuring NetFlow Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style CLI Example: In the following commands, the destination endpoint group is the endpoint group that the exporter sits behind, which in this case is an external Layer 3 endpoint group. apic1(config)# flow exporter tnExporter2 apic1(config-flow-exporter)# transport udp 9990 apic1(config-flow-exporter)# destination address 2001:db5:a0c:1f0::2 apic1(config-flow-exporter)# destination external-l3 epg tenant tn2 vrf v2 epg accounting-inst apic1(config-flow-exporter)# vrf member tenant tn2 vrf vrf2 apic1(config-flow-exporter)# version v5 apic1(config-flow-exporter)# source address 2001:db8:a0b:12f0::1 apic1(config-flow-exporter)# exit Step 8 Create a NetFlow record policy. Example: apic1(config)# flow record apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# apic1(config-flow-record)# Step 9 tnRecord1 match dst-ip match dst-ipv4 match dst-ipv6 match dst-mac match dst-port match ethertype match proto match src-ip match src-ipv4 match src-ipv6 match src-mac match src-port match tos match vlan collect count-bytes collect count-pkts collect pkt-disp collect sampler-id collect src-intf collect tcp-flags collect ts-first collect ts-recent exit Create a NetFlow monitor policy. Example: apic1(config)# flow monitor apic1(config-flow-monitor)# apic1(config-flow-monitor)# apic1(config-flow-monitor)# apic1(config-flow-monitor)# tnMonitor1 record tnRecord1 exporter tnExporter1 exporter tnExporter2 exit You can attach a maximum of two exporters. Step 10 Add VLANs to the VLAN domain and configure a VRF for a leaf node. Example: apic1(config)# vlan-domain dom1 apic1(config-vlan)# vlan 5-100 apic1(config-vlan)# exit apic1(config)# leaf 101 apic1(config-leaf)# vrf context tenant tn2 vrf vrf2 apic1(config-leaf-vrf)# exit Step 11 Deploy an endpoint group on an interface to deploy the bridge domain. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 350 Configuring NetFlow Consuming a NetFlow Exporter Policy Under a VMM Domain Using the NX-OS-Style CLI for VMware VDS Example: apic1(config-leaf)# interface ethernet 1/10 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant tn2 application ap2 epg epg2 apic1(config-leaf-if)# exit Step 12 Deploy another endpoint group on an interface. Example: apic1(config-leaf)# interface ethernet 1/11 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk allowed vlan 11 tenant tn2 application ap3 epg epg3 apic1(config-leaf-if)# exit Step 13 Attach the monitor policy to the sub-interface. Example: apic1(config-leaf)# interface ethernet 1/20 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# no switchport apic1(config-leaf-if)# exit apic1(config-leaf)# interface ethernet 1/20.20 apic1(config-leaf-if)# vrf member tenant tn2 vrf vrf2 apic1(config-leaf-if)# ipv6 address 20::1/64 preferred apic1(config-leaf-if)# ipv6 flow monitor tnMonitor1 apic1(config-leaf-if)# ip flow monitor tnMonitor2 apic1(config-leaf-if)# exit Step 14 Attach the monitor policy to a switched virtual interface (SVI). Example: apic1(config-leaf)# interface vlan 30 apic1(config-leaf-if)# vrf member tenant tn2 vrf vrf2 apic1(config-leaf-if)# ipv6 address 64::1/64 preferred apic1(config-leaf-if)# ip flow monitor tnMonitor1 apic1(config-leaf-if)# ip6 flow monitor tnMonitor1 apic1(config-leaf-if)# exit Step 15 Associate the SVI to a Layer 2 interface. Example: apic1(config-leaf)# interface ethernet 1/30 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk allowed vlan 30 tenant tn2 external-svi apic1(config-leaf-if)# exit apic1(config-leaf)# exit apic1(config)# exit Consuming a NetFlow Exporter Policy Under a VMM Domain Using the NX-OS-Style CLI for VMware VDS The following procedure uses the NX-OS-style CLI to consume a NetFlow exporter policy under a VMM domain. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 351 Configuring NetFlow Enabling or Disabling NetFlow on an Endpoint Group Using the NX-OS-Style CLI for VMware VDS Procedure Step 1 Enter the configuration mode. Example: apic1# config Step 2 Consume the NetFlow exporter policy. Example: apic1(config)# vmware-domain mininet apic1(config-vmware)# configure-dvs apic1(config-vmware-dvs)# flow exporter apic1(config-vmware-dvs-flow-exporter)# apic1(config-vmware-dvs-flow-exporter)# apic1(config-vmware-dvs-flow-exporter)# apic1(config-vmware-dvs-flow-exporter)# apic1(config-vmware-dvs)# exit apic1(config-vmware)# exit apic1(config)# exit vmExporter1 active-flow-timeout 62 idle-flow-timeout 16 sampling-rate 1 exit Enabling or Disabling NetFlow on an Endpoint Group Using the NX-OS-Style CLI for VMware VDS The following procedure enables or disables NetFlow on an endpoint group using the NX-OS-style CLI. Procedure Step 1 Enable NetFlow: Example: apic1# config apic1(config)# tenant tn1 apic1(config-tenant)# application app1 apic1(config-tenant-app)# epg epg1 apic1(config-tenant-app-epg)# vmware-domain member mininet apic1(config-tenant-app-epg-domain)# flow monitor enable apic1(config-tenant-app-epg-domain)# exit apic1(config-tenant-app-epg)# exit apic1(config-tenant-app)# exit apic1(config-tenant)# exit apic1(config)# exit Step 2 (Optional) If you no longer want to use NetFlow, disable the feature: Example: apic1(config-tenant-app-epg-domain)# no flow monitor enable Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 352 CHAPTER 14 Managing Firmware • Managing Firmware, page 353 • Adding or Removing Repository Images, page 354 • Changing Catalog Firmware, page 354 • Upgrading Controller Firmware, page 355 • Upgrading Switch Firmware, page 357 Managing Firmware Each firmware image includes a compatibility catalog that identifies supported types and switch models. APIC maintains a catalog of the firmware images, switch types, and models that are allowed to use that firmware image. The default setting is to reject a firmware update when it does not conform to the compatibility catalog. APIC has an image repository for compatibility catalogs, controller firmware images, and switch images. The administrator can download new firmware image to the APIC image repository from an external HTTP server or SCP server. Note Before you upgrade the switches, the APICs must have completed upgrading and have a health state of Fully Fit. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 353 Managing Firmware Adding or Removing Repository Images Adding or Removing Repository Images Procedure Step 1 Command or Action Purpose firmware repository add absolute-image-path Adds a firmware image to the repository. Example: apic1# firmware repository add /home/admin/aci-catalog-dk9.1.2.1b.bin Step 2 firmware repository delete image Deletes a firmware image from the repository. Example: apic1# firmware repository delete aci-catalog-dk9.1.2.1a.bin Examples apic1# firmware repository add /home/admin/aci-catalog-dk9.1.2.1b.bin apic1# firmware repository delete aci-catalog-dk9.1.2.1a.bin Changing Catalog Firmware This procedures shows how to select a catalog firmware version from the repository. Procedure Step 1 Command or Action Purpose show firmware repository [detail] apic1# show firmware repository Show firmware images present in repository. The detail option displays additional information such as MD5 checksum, release date, and download date. configure Enters global configuration mode. Example: Step 2 Example: apic1# configure Step 3 firmware Example: apic1(config)# firmware Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 354 Enters firmware upgrade configuration mode. Managing Firmware Upgrading Controller Firmware Step 4 Command or Action Purpose show version (Optional) Displays the currently-installed controller and switch firmware versions. Example: apic1(config-firmware)# show version Step 5 catalog-version firmware-name Changes the catalog version to an available image in the repository. Example: apic1(config-firmware)# catalog-version aci-catalog-dk9.1.2.1b.bin Examples This example shows how to select a catalog firmware version from the repository. apic1# show firmware repository Name Type -------------------------- ------aci-catalog-dk9.1.2.1a.bin catalog aci-catalog-dk9.1.2.1b.bin catalog Version ------1.2.1a 1.2.1b Size(MB) ------0.023 0.025 apic1# configure apic1(config)# firmware apic1(config-firmware)# catalog-version aci-catalog-dk9.1.2.1b.bin Upgrading Controller Firmware The controllers upgrade in random order. Each APIC controller takes about 10 minutes to upgrade. Once a controller image is upgraded, it drops from the cluster and reboots with the newer version while the other APIC controllers in the cluster are still operational. Once the controller reboots, it joins the cluster again. Then the cluster converges, and the next controller image starts to upgrade. The catalog firmware image is upgraded when an APIC controller image is upgraded. You do not need to upgrade the catalog firmware image separately. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 Enters firmware upgrade configuration mode. firmware Example: apic1(config)# firmware Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 355 Managing Firmware Upgrading Controller Firmware Step 3 Command or Action Purpose show version (Optional) Displays the currently-installed controller and switch firmware versions. Example: apic1(config-firmware)# show version Step 4 Enters controller upgrade configuration mode. controller-group Example: apic1(config-firmware)# controller-group Step 5 firmware-version firmware-name Specifies the desired version for the upgrade. Example: apic1(config-firmware-controller)# firmware-version aci-apic-dk9.1.2.1b.iso Step 6 [no] time start time Example: Sets the starting time in the format [[[yyyy:]mmm:]dd:]HH:MM. The date is optional. apic1(config-firmware-controller)# time start 2016:jan:01:12:01 Note To upgrade the controllers immediately, return to EXEC mode and type the command firmware upgrade controller-group. Examples This example shows how to upgrade the controllers. apic1# show controller Fabric Name : Operational Size : Cluster Size : Time Difference : Fabric Security Mode : mininet 3 3 0 permissive ID Address In-Band Address OOB Address Version Flags Serial Number --- --------- --------------- ------------ ---------- ----- ------------- --------- 1* 10.0.0.1 192.168.11.1 192.168.10.1 1.2(1a) crva TEP-1-1 fully-fit 2 10.0.0.2 192.168.11.2 192.168.10.2 1.2(1a) crva TEP-1-2 fully-fit 3 10.0.0.3 192.168.11.3 192.168.10.3 1.2(1a) crva TEP-1-3 fully-fit Flags - c:Commissioned | r:Registered | v:Valid Certificate | a:Approved apic1# configure apic1(config)# firmware apic1(config-firmware)# show version Role Id Name ---------- ---------- ----------------controller 1 apic1 controller 2 apic2 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 356 Version ----------1.2(1a) 1.2(1a) Health Managing Firmware Upgrading Switch Firmware controller leaf leaf leaf spine spine 3 101 102 103 201 202 apic3 leaf1 leaf2 leaf2 spine1 spine2 1.2(1a) n9000-11.2(1a) n9000-11.2(1a) n9000-11.2(1a) n9000-11.2(1a) n9000-11.2(1a) apic1(config-firmware)# controller-group apic1(config-firmware-controller)# firmware-version aci-apic-dk9.1.2.1b.iso apic1(config-firmware-controller)# time start 2016:jan:01:12:01 Upgrading Switch Firmware Before You Begin A scheduler must exist to specify when the upgrade will be executed. Note Before you upgrade the switches, the APICs must have completed upgrading and have a health state of Fully Fit. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 Enters firmware upgrade configuration mode. firmware Example: apic1(config)# firmware Step 3 [no] switch-group group-name Creates (or deletes) switch group and enters switch upgrade configuration mode. Example: apic1(config-firmware)# switch-group mySwitchGroup5 Step 4 [no] switch node-id-or-name[,node-id-or-name,...] Example: Adds (or removes) a switch or a list of switches to the switch-group for upgrading. You can specify the node ID (such as 101) or the name (such as spine1). You can specify multiple switches by using commas. apic1(config-firmware-switch)# switch leaf1-leaf3,leaf6 apic1(config-firmware-switch)# no switch leaf4,leaf5 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 357 Managing Firmware Upgrading Switch Firmware Step 5 Command or Action Purpose firmware-version firmware-name Specifies the target firmware image. Example: apic1(config-firmware-switch)# firmware-version aci-apic-dk9.11.2.1a.bin Step 6 [no] run-mode {pause-never | pause-on-failure} Species whether to proceed to the next set of nodes if the upgrade fails on the current set of nodes. Example: apic1(config-firmware-switch)# run-mode pause-on-failure Step 7 schedule scheduler-name Example: Assigns a scheduler for the upgrade. Enter the name of a scheduler that has already been defined. apic1(config-firmware-switch)# schedule Note myNextSunday Step 8 [no] scheduler pause To upgrade the switch group immediately, return to EXEC mode and type the command firmware upgrade switch-group. Pauses the maintenance policy scheduler. Use the [no] prefix to resume. Example: apic1(config-firmware-switch)# scheduler pause apic1(config-firmware-switch)# no scheduler pause Step 9 show running-config Displays the configuration. Example: apic1(config-firmware-switch)# show run Examples This example shows how to upgrade the firmware for three leaf switches. apic1# configure apic1(config)# firmware apic1(config-firmware)# switch-group mySwitchGroup5 apic1(config-firmware-switch)# switch leaf1,leaf3,leaf6 apic1(config-firmware-switch)# no switch leaf4,leaf5 apic1(config-firmware-switch)# firmware-version aci-apic-dk9.1.1.3f.bin apic1(config-firmware-switch)# run-mode pause-on-failure apic1(config-firmware-switch)# schedule myNextSunday apic1(config-firmware-switch)# show run # Command: show running-config firmware switch-group mySwitchGroup5 # Time: Fri Nov 6 23:55:35 2015 firmware switch-group mySwitchGroup5 switch 101 switch 102 switch 103 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 358 Managing Firmware Upgrading Switch Firmware switch 106 schedule myNextSunday exit exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 359 Managing Firmware Upgrading Switch Firmware Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 360 CHAPTER 15 Managing the Configuration with Snapshots • About Configuration Management and Snapshots, page 361 • Exporting a Snapshot, page 361 • Importing a Snapshot, page 363 • Rollback Configuration Using Snapshots, page 364 • Uploading or Downloading a Snapshot File to a Remote Path, page 366 • Managing Snapshot Files and Jobs, page 367 About Configuration Management and Snapshots You can back up and restore your system configuration by exporting and importing configuration archives (snapshots) to and from a local controller-managed folder. By exporting snapshots before and after making configuration changes, you have the ability to roll back configuration changes that were applied between two snapshots. You can also upload and download the snapshot files to and from a remote server. Each snapshot action (export, import, rollback, upload, and download) is performed by creating a policy for the action and then triggering the action as a job. Export actions can also be scheduled to run at a future time or periodically. Import, export, and rollback jobs cannot run in parallel. If a job is already running, triggering a new job will fail. Exporting a Snapshot Before You Begin If you want to export snapshots according to a schedule, configure a scheduler before configuring the export policy. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 361 Managing the Configuration with Snapshots Exporting a Snapshot Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] snapshot export policy-name Creates a policy for exporting snapshots. Example: apic1(config)# snapshot export myExportPolicy Step 3 format {xml | json} Specifies the data format for the exported configuration file. The default is Example: apic1(config-export)# format json Step 4 [no] schedule schedule-name Example: (Optional) Specifies an existing scheduler for exporting snapshots. apic1(config-export)# schedule EveryEightHours Step 5 [no] target [infra | fabric | tenant-name] Example: apic1(config-export)# target tenantExampleCorp Step 6 [no] remote path remote-path-name apic1(config-export)# remote path myBackupServer (Optional) Specifies the name of a configured remote path to which the file will be sent. If no remote path is specified, the file is exported locally to a folder in the controller. The default is no remote path. end Returns to EXEC mode. Example: Step 7 (Optional) Assigns the target of the export, which can be fabric, infra, a specific tenant, or none. If no target is specified, all configuration information is exported. The default is no target. Example: apic1(config-export)# end Step 8 trigger snapshot export policy-name Example: apic1# trigger snapshot export myExportPolicy Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 362 Executes the snapshot export task. If the export policy is configured with a scheduler, this step is unnecessary unless you want an immediate export. Managing the Configuration with Snapshots Importing a Snapshot Examples This example shows how to configure the periodic export of a JSON-format snapshot file for a specific tenant configuration. apic1# configure apic1(config)# snapshot export myExportPolicy apic1(config-export)# format json apic1(config-export)# target tenantExampleCorp apic1(config-export)# schedule EveryEightHours Importing a Snapshot Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] snapshot import policy-name Creates a policy for importing snapshots. Example: apic1(config)# snapshot import myImportPolicy Step 3 file filename Specifies the name of the file to be imported. Example: apic1(config-import)# file ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz Step 4 action {merge | replace} Example: apic1(config-import)# action replace Step 5 [no] mode {atomic | best-effort} Example: apic1(config-import)# mode atomic Step 6 [no] remote path remote-path-name Example: apic1(config-import)# remote path myBackupServer Specifies whether the imported configuration settings will be merged with the current settings or whether the imported configuration will completely replace the current configuration. Specifies how the import process handles configuration errors when applying the imported settings. The best-effort import mode allows skipping individual configuration errors in the archive, while atomic mode cancels the import upon any configuration error. (Optional) Specifies the name of a configured remote path from which the file will be imported. If no remote path is specified, the file is Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 363 Managing the Configuration with Snapshots Rollback Configuration Using Snapshots Command or Action Purpose imported locally from a folder in the controller. The default is no remote path. Step 7 end Returns to EXEC mode. Example: apic1(config-import)# end Step 8 trigger snapshot import policy-name Executes the snapshot import task. Example: apic1# trigger snapshot import myImportPolicy Examples This example shows how to configure and execute the importing of a snapshot file to replace the current configuration. apic1# show snapshot files File : ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz Created : 2015-11-21T01:00:21.167+00:00 Root : Size : 22926 apic1# configure apic1(config)# snapshot import myImportPolicy apic1(config-import)# file ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz apic1(config-import)# action replace apic1(config-import)# mode atomic apic1(config-import)# end apic1# trigger snapshot import myImportPolicy Rollback Configuration Using Snapshots The rollback feature provides an "undo" function that reverts changes made between one snapshot archive and a later snapshot archive. Only locally stored snapshot files are supported for rollback. You can optionally enable the preview mode to generate and view a rollback before implementing it. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 364 Managing the Configuration with Snapshots Rollback Configuration Using Snapshots Step 2 Command or Action Purpose [no] snapshot rollback policy-name Creates a policy for rollback using snapshots. Example: apic1(config)# snapshot rollback myRollbackPolicy Step 3 first-file filename Specifies the name of the earlier file. Example: apic1(config-rollback)# first-file ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz Step 4 second-file filename Specifies the name of the later file. Example: apic1(config-rollback)# second-file ce2_DailyAutoBackup-2015-11-21T09-00-21.tar.gz Step 5 [no] preview Example: apic1(config-rollback)# preview Step 6 (Optional) Specifies that the rollback changes are generated and previewed but not applied. When preview mode is enabled, no changes to the configuration are made. After previewing rollback changes, use the no preview command to exit preview mode and enable the rollback to be applied when you reenter the trigger snapshot rollback commands. Returns to EXEC mode. end Example: apic1(config-rollback)# end Step 7 trigger snapshot rollback policy-name Executes the snapshot rollback task. Example: apic1# trigger snapshot rollback myRollbackPolicy Examples This example shows how to configure and execute a rollback without previewing it first. apic1# show snapshot files File : ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz Created : 2015-11-21T01:00:21.167+00:00 Root : Size : 22926 File Created Root Size : ce2_DailyAutoBackup-2015-11-21T09-00-21.tar.gz : 2015-11-21T09:00:24.025+00:00 : : 23588 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 365 Managing the Configuration with Snapshots Uploading or Downloading a Snapshot File to a Remote Path apic1# configure apic1(config)# snapshot apic1(config-rollback)# apic1(config-rollback)# apic1(config-rollback)# apic1# trigger snapshot rollback myRollbackPolicy first-file ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz second-file ce2_DailyAutoBackup-2015-11-21T09-00-21.tar.gz end rollback myRollbackPolicy Uploading or Downloading a Snapshot File to a Remote Path You can upload snapshot archive files from local storage to a remote path. You can also download snapshot archive files from the remote path. Before You Begin You must configure a remote path to receive the file. See Configuring a Remote Path for File Export, on page 381. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] snapshot {upload | download} policy-name remote-path-name Creates a policy for uploading or downloading snapshot files with a remote path. Example: apic1(config)# snapshot upload myUpPolicy Step 3 remote path remote-path-name Example: Specifies the name of a configured remote path to which the snapshot file will be sent. apic1(config-upload)# remote path myBackupServer Step 4 file filename Specifies the name of the snapshot file to be sent. Example: apic1(config-upload)# file ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz Step 5 end Returns to EXEC mode. Example: apic1(config-upload)# end Step 6 trigger snapshot {upload | download} policy-name Example: apic1# trigger snapshot upload myUpPolicy Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 366 Executes the snapshot upload or download task. Managing the Configuration with Snapshots Managing Snapshot Files and Jobs Examples This example shows how to configure and execute the uploading of a snapshot file to a remote path. apic1# show snapshot files File : ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz Created : 2015-11-21T01:00:21.167+00:00 Root : Size : 22926 apic1# configure apic1(config)# snapshot upload myUpPolicy apic1(config-upload)# remote path myBackupServer apic1(config-upload)# file ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz apic1(config-upload)# end apic1# trigger snapshot upload myUpPolicy Managing Snapshot Files and Jobs The following commands are available for managing snapshot files and jobs. Command Description clear snapshot file filename Removes a snapshot file from the local storage. clear snapshot job job-name Removes a snapshot job from the history. show snapshot files Displays the snapshot files in local storage. show snapshot jobs Displays recent snapshot tasks. show snapshot active jobs Displays currently-active snapshot tasks. Examples This example shows how to display snapshot files and the snapshot job history. apic1# show snapshot files File : ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz Created : 2015-11-21T01:00:21.167+00:00 Root : Size : 22926 File Created Root Size : ce2_DailyAutoBackup-2015-11-21T09-00-21.tar.gz : 2015-11-21T09:00:24.025+00:00 : : 23588 apic1# show Type : Run : State : Details : File Name : snapshot jobs export 2015-11-21T01-00-17 success Success ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 367 Managing the Configuration with Snapshots Managing Snapshot Files and Jobs Type Run State Details File Name : : : : : export 2015-11-21T09-00-21 success Success ce2_DailyAutoBackup-2015-11-21T09-00-21.tar.gz Type Run State Details File Name : : : : : rollback 2015-11-22T00-25-06 running not applicable apic1# clear snapshot job 2015-11-22T00-25-06 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 368 CHAPTER 16 Configuring Monitoring • Configuring Syslog, page 369 • Configuring Call Home, page 372 • Sending an On-Demand Techsupport File Using the NX-OS Style CLI, page 380 • Configuring a Remote Path for File Export, page 381 • Using Show Commands for Monitoring, page 382 • Configuring SNMP, page 388 Configuring Syslog Configuring a Logging Server Group In the ACI fabric, one or more logging server-groups can be configured with one or more logging destination servers. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 logging server-group server-group-name Configure a grouping of servers for monitoring. Example: apic1(config)# logging server-group myLoggingGroup Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 369 Configuring Monitoring Configuring a Logging Server Group Step 3 Command or Action Purpose [no] description text Specifies Example: apic1(config-logging)# logging description "This is my logging server group" Step 4 [no] console [severity {alerts | critical | emergencies}] Enables logging to the console (only for switches) and optionally sets the minimum severity level for logging. Example: apic1(config-logging)# console severity critical Step 5 [no] logfile [severity {alerts | critical | Enables logging to the logfile and optionally debugging | emergencies | errors | information sets the minimum severity level for logging. | notifications | warnings}] Example: apic1(config-logging)# logfile severity critical Step 6 Adds a destination logging server and optionally [no] server ip-address-or-hostname [facility local-level] [severity severity-level] [mgmtepg sets the minimum severity level for logging. {inb | oob}] [port port-number] • facility—Local facility in the form localn Example: apic1(config-logging)# server reach.example.com level local4 mgmtepg inb port 514 • severity—Minimum severity level for logging. Can be one of the options shown in the logfile command. • mgmt—Management endpoint group, either inb (inband) or oob (out of band). • port—Service port number of the logging server. Examples This example shows how to configure a syslog destination server group. apic1# configure apic1(config)# logging apic1(config-logging)# apic1(config-logging)# apic1(config-logging)# apic1(config-logging)# server-group myLoggingGroup logging description "This is my logging server group" console severity critical logfile severity critical server reach.example.com level local4 mgmtepg inb port 514 What to Do Next Configure syslog with this logging server group as the logging destination. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 370 Configuring Monitoring Configuring Syslog Configuring Syslog In order to receive and monitor system log messages, you must specify a syslog destination, which can be the console, a local file, or one or more remote hosts running a syslog server. In addition, you can specify the minimum severity level of messages to be displayed on the console or captured by the file or host. Before You Begin Configure a logging server-group containing the servers to which syslog messages will be sent. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 Enters syslog common policy configuration mode. syslog common Example: apic1(config)# syslog common Step 3 [no] logging description text Adds descriptive text about the policy. Example: apic1(config-syslog)# logging description "This is the common logging policy" Step 4 [no] logging severity {alerts | critical | debugging | Specifies the minimum severity level emergencies | errors | information | notifications | for sending syslog messages. warnings} Example: apic1(config-syslog)# logging severity notifications Step 5 [no] logging server-group server-group-name Specifies a destination logging server group. Example: apic1(config-syslog)# logging server-group myLoggingGroup Step 6 Enables audit logs to the policy. [no] logging audit Example: apic1(config-syslog)# logging audit Step 7 Enables event logs to the policy. [no] logging event Example: apic1(config-syslog)# logging event Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 371 Configuring Monitoring Configuring Call Home Step 8 Command or Action Purpose [no] logging fault Enables fault logs to the policy. Example: apic1(config-syslog)# logging fault Step 9 [no] logging session Enables session logs to the policy. Example: apic1(config-syslog)# logging session Examples This example shows how to configure syslog for messages of 'notification' severity or higher. Syslog messages from fault and event logs are sent to servers in server-group myLoggingGroup. apic1# configure apic1(config)# syslog apic1(config-syslog)# apic1(config-syslog)# apic1(config-syslog)# apic1(config-syslog)# apic1(config-syslog)# common logging logging logging logging logging description "This is the common logging policy" severity notifications server-group myLoggingGroup audit event Configuring Call Home Configuring the Call Home Policy In the ACI fabric, Cisco Call Home configuration can be added in the common monitoring policy. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 callhome common Enters Call Home common policy configuration mode. Example: apic1(config)# callhome common Step 3 [no] logging audit Example: apic1(config-callhome)# logging audit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 372 Enables audit logs to the policy. Configuring Monitoring Configuring the Call Home Policy Step 4 Command or Action Purpose [no] logging event Enables event logs to the policy. Example: apic1(config-callhome)# logging event Step 5 Enables fault logs to the policy. [no] logging fault Example: apic1(config-callhome)# logging fault Step 6 [no] logging severity {alert | critical | debug | emergency | error | info | notice | warning} Specifies the minimum severity level for logging. Example: apic1(config-callhome)# logging severity notice Step 7 [no] periodic-inventory notification schedule scheduler Configures a periodic notification scheduler. The scheduler must be previously configured. Example: apic1(config-callhome)# periodic-inventory notification schedule EveryEightHours Step 8 show callhome common [destination-profile | query-profile | transport-email] Shows Call Home configuration. Example: apic1(config-callhome)# show callhome common Examples This example shows how to configure a basic Call Home policy. apic1# configure apic1(config)# callhome common apic1(config-callhome)# logging event apic1(config-callhome)# logging fault apic1(config-callhome)# logging severity notice apic1(config-callhome)# periodic-inventory notification schedule EveryEightHours apic1(config-callhome)# end apic1# show callhome common Callhome : common Logging Enabled : event,faults Logging Severity : notice Destination-Profile : Admin State Contract-id Customer-id Email-addr From email-addr Reply-To email-addr Phone Number : : : : : : : Enabled 12345678 ABCDEFG [email protected] [email protected] [email protected] +14085551212 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 373 Configuring Monitoring Configuring a Call Home Destination Profile SMTP Port num : 25 SMTP Server : smtp.example.com Destination ----------SanJose Email-addr [email protected] Format -----xml Message-Size -----------40000 Message-Level ------------alert Query-Profile : Query Name Query Type Dn/Class -------------------------------------------------------myUserQuery class User ep-records,fault-records,stats Target ------ Respones Subtree ---------------- self children Response Include What to Do Next Configure a destination profile and (optionally) a query profile. Configuring a Call Home Destination Profile You must configure at least one destination profile for Call Home. If the destination profile uses email message delivery, you must specify a Simple Mail Transfer Protocol (SMTP) server. Before You Begin Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 callhome common Enters Call Home common policy configuration mode. Example: apic1(config)# callhome common Step 3 [no] destination-profile Configures a destination profile. Example: apic1(config-callhome)# destination-profile Step 4 [no] destination dest-name Example: Configures a destination where the Call Home messages will be sent, including the format of the messages and the severity level for sending. apic1(config-callhome-destnprof)# destination SanJose Note Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 374 You can configure more than one destination. Configuring Monitoring Configuring a Call Home Destination Profile Step 5 Command or Action Purpose [no] email-addr email Configures the e-mail address that will receive the Call Home messages. Up to 255 alphanumeric characters are accepted in e-mail address format. Example: apic1(config-callhome-destnprof-destn)# email-addr [email protected] Step 6 [no] format {aml | xml | short-txt} Example: apic1(config-callhome-destnprof-destn)# format xml Configures the format for Call Home messages, which can be sent in the following formats: • aml—Adaptive Messaging Language (AML) XML schema definition (XSD) • xml—The XML format enables communication with the Cisco Systems Technical Assistance Center (TAC). • short-txt—Short text format provides a one or two line description of the fault that is suitable for pagers or printed reports. Step 7 [no] message-level {alert | critical | debug | Configures the minimum severity level for emergency | error | info | notice | warning} sending messages. Example: apic1(config-callhome-destnprof-destn)# message-level alert Step 8 [no] message-size size Configures the size of the messages. The range is 0 to 5000000 characters. Example: apic1(config-callhome-destnprof-destn)# message-size 40000 Step 9 Returns to destination profile configuration mode. exit Example: apic1(config-callhome-destnprof-destn)# exit Step 10 Configure the destination profile. Use the commands in Call Home Destination Profile Configuration Commands, on page 376 Example: apic1(config-callhome-destnprof)# (various commands) Step 11 show callhome common [destination-profile Shows Call Home configuration. | query-profile | transport-email] Example: apic1(config-callhome-destnprof)# show callhome common transport-email Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 375 Configuring Monitoring Configuring a Call Home Destination Profile Examples This example shows how to configure Call Home to send email messages of severity 'alert' or higher to [email protected] apic1# configure apic1(config)# callhome common apic1(config-callhome)# destination-profile apic1(config-callhome-destnprof)# destination SanJose apic1(config-callhome-destnprof-destn)# email-addr [email protected] apic1(config-callhome-destnprof-destn)# format xml apic1(config-callhome-destnprof-destn)# message-level alert apic1(config-callhome-destnprof-destn)# message-size 40000 apic1(config-callhome-destnprof-destn)# exit apic1(config-callhome-destnprof)# contract-id 12345678 apic1(config-callhome-destnprof)# customer-id ABCDEFG apic1(config-callhome-destnprof)# description "Example Corporation" apic1(config-callhome-destnprof)# site-id XYZ123 apic1(config-callhome-destnprof)# street-address "1 Cisco Way" apic1(config-callhome-destnprof)# phone-contact +14085551212 apic1(config-callhome-destnprof)# email-contact [email protected] apic1(config-callhome-destnprof)# transport email from [email protected] apic1(config-callhome-destnprof)# transport email reply-to [email protected] apic1(config-callhome-destnprof)# transport email mail-server smtp.example.com mgmtepg inb port 25 apic1(config-callhome)# end apic1# show callhome common transport-email From email-addr : [email protected] SMTP Port num : 25 SMTP Server : smtp.example.com Call Home Destination Profile Configuration Commands These commands are entered in the Call Home destination profile (config-callhome-destnprof) configuration mode. Command Purpose contract-id contract-id The Call Home contract number for the customer. customer-id customer-id The CCO ID that includes the contract numbers for the support contract in its entitlements. description text Descriptive text about this customer site. email-contact email The email address for the main contact. phone-contact phone-num The telephone number for the main contact. site-id site-id The unique Call Home identification number for the customer site. street-address address The mailing address for the main contact. transport email from email The email address that should appear in the From field on Call Home alert messages sent by the system. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 376 Configuring Monitoring Configuring a Call Home Query transport email reply-to email The return email address that should appear in the From field on Call Home alert messages sent by the system. transport email mail-server smtp-server mgmtepg The IP address or hostname of the SMTP server and the port number the system should use to talk to the {inb | oob} port port-number SMTP server. Configuring a Call Home Query When an event triggers the sending of a Call Home report, information from your selected queries is included in the report. You can configure a query based on a class name or a distinguished name, and you can further qualify the query based on subtrees. Before You Begin Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 Enters Call Home common policy configuration mode. callhome common Example: apic1(config)# callhome common Step 3 Enters Call Home query profile configuration mode. [no] query-profile Example: apic1(config-callhome)# query-profile Step 4 [no] query query-name type {class class-name | Configures a query profile. dn name} Example: apic1(config-callhome-queryprof)# query myUserQuery type class User Step 5 [no] response-subtree {full | children | no} Example: Configures the response subtree. You can choose to include the full subtree, only children, or no subtree information. apic1(config-callhome-queryprof-query)# response-subtree children Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 377 Configuring Monitoring Configuring a Call Home Query Step 6 Command or Action Purpose [no] response-incl {option[,option[,option...]]} Configures the specific subtree information categories to be included in the response. Multiple categories can be specified in a comma-separated list. The available categories are listed in Query Subtree Categories, on page 378. Example: apic1(config-callhome-queryprof-query)# response-incl ep-records,fault-records,stats Step 7 [no] target {children | self | subtree} Configures the query target. Example: apic1(config-callhome-queryprof-query)# target self Step 8 show callhome common [destination-profile | query-profile | transport-email] Shows Call Home configuration. Example: apic1(config-callhome-queryprof-query)# show callhome common query-profile Examples This example shows how to configure a Call Home query. apic1# configure apic1(config)# callhome common apic1(config-callhome)# query-profile apic1(config-callhome-queryprof)# query myUserQuery type class User apic1(config-callhome-queryprof-query)# response-subtree children apic1(config-callhome-queryprof-query)# response-incl ep-records,fault-records,stats apic1(config-callhome-queryprof-query)# target self apic1(config-callhome)# end apic1# show callhome common destination-profile Query-Profile : Query Name Query Type Dn/Class -------------------------------------------------------myUserQuery class User ep-records,fault-records,stats Target ------ Respones Subtree ---------------- self children Query Subtree Categories Query Category Description add-mo-list audit-logs config-only Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 378 Response Include Configuring Monitoring Configuring a Call Home Query count custom-path-hop deployment deployment-records ep-records event-logs fault-count fault-records faults full-deployment health health-records local-prefix no-scoped none port-deployment record-subtree relations relations-with-parent required state stats tasks Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 379 Configuring Monitoring Sending an On-Demand Techsupport File Using the NX-OS Style CLI Sending an On-Demand Techsupport File Using the NX-OS Style CLI Note Do not trigger techsupport file collection from more than five nodes simultaneously, especially if they are to be exported into the APIC or to an external server with insufficient bandwidth and compute resources. To avoid excessive storage usage in APIC, remove locally-stored techsupport files promptly. Before You Begin Configure a remote path for exporting the techsupport file. Procedure Step 1 Command or Action Purpose trigger techsupport {all | controllers switch node-id} [remotename remote-path-name] Triggers the export of a techsupport file from the controllers, switches, or all to the remote path. For switches, you can specify a range or a comma-separated list. If no remote host is specified, the file is collected in the controller itself. Example: apic1# trigger techsupport switch 101,103 remotename remote5 Step 2 trigger techsupport host host-id Example: Triggers the export of a techsupport file from the specified host to the remote host. If no remote host is specified, the file is collected in the controller itself. apic1# trigger techsupport host Step 3 trigger techsupport local Example: Triggers the export of a local techsupport file to the remote host. If no remote host is specified, the file is collected in the controller itself. apic1# trigger techsupport local Step 4 show techsupport {all | controllers switch After a techsupport file is triggered, this command shows the status of the techsupport report. node-id} status Example: apic1# show techsupport switch 101 status Examples This example shows how to trigger a techsupport file for switch 101, to be stored locally on the apic1 controller. apic1# trigger techsupport switch 101 Triggering techsupport for Switch 101 using policy supNode101, setting filters to default value Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 380 Configuring Monitoring Configuring a Remote Path for File Export Triggered on demand tech support successfully for Switch 101, will be available at: /data/techsupport on the controller. Use 'show techsupport' with your options to check techsupport status. Configuring a Remote Path for File Export In the ACI fabric, you can configure one or more remote destinations for exporting techsupport or configuration files. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] remote path remote-path-name Enters configuration mode for a remote path. Example: apic1(config)# remote path myFiles Step 3 user username Sets the user name for logging in to the remote server. You are prompted for a password. Example: apic1(config-remote)# user admin5 Step 4 path {ftp | scp | sftp} host[ :port ] [remote-directory ] Sets the path and protocol to the remote server. You are prompted for a password. Example: apic1(config-remote)# path sftp filehost.example.com:21 remote-directory /reports/apic Examples This example shows how to configure a remote path for exporting files. apic1# configure apic1(config)# remote path myFiles apic1(config-remote)# user admin5 You must reset the password when modifying the path: Password: Retype password: apic1(config-remote)# path sftp filehost.example.com:21 remote-directory /reports/apic You must reset the password when modifying the path: Password: Retype password: Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 381 Configuring Monitoring Using Show Commands for Monitoring Using Show Commands for Monitoring About Using the Show Commands The show commands for faults, events, health, statistics, and audit logs can be filtered to display specific types of information or information from specific entities, such as controllers, leaf switches, spine switches, or tenants. Broad queries are expensive in terms of system resources and storage. For example, using the show faults, events, or audit commands without entity filters retrieves all logs or records from the entire system. We recommend that you make use of the available data and entity filters to narrow your query as much as possible. For example, the following command would result in a quicker and more filtered response by limiting the query to the most recent 45 minute period: show audits last-minutes 45 Tip At each point in the command, typing ‘ ?’ displays all possible keywords and options that can be used at that point along with a brief explanation of each. Using the show faults Command The show faults command can combine several data filters and an entity filter to deliver a specific set of faults. The command syntax is: show faults [filter1 [filter2... ]] [entity-filter] Entity filters restrict the output to faults of a controller, leaf, spine, or tenant. The available entity filters are listed in Entity Filters for Show Commands, on page 387. Data filters are provided to make the task of querying faults easier for the user. The available data filters are: Filter Description ack {yes | no} acknowledgment status cause name cause code fault-code fault code controller controller information detail detailed faults information end-time YYYY-MM-DDTHR-MM:SS fault activity up to this time history historical information id fault-id fault ID Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 382 Configuring Monitoring Using the show faults Command Filter Description l4l7-cluster[cluster name | tenant name] L4 L7 device information l4l7-graph[cluster name | tenant name] L4 L7 graph information last-days days fault activity in the last N days last-hours hours fault activity in the last N hours last-minutes minutes fault activity in the last N minutes lc lc-state lifecycle state leaf [leaf-id] leaf switch information microsoft domain name Microsoft domain information min-severity severity-value minimum severity severity severity-value severity spine [spine-id] spine switch information start-time YYYY-MM-DDTHR-MM:SS fault activity starting from this time tenant [name] tenant information type fault-type fault type vmware domain name VMware domain information Examples This example shows all faults that occurred in the past five days with code “F110473”, severity “warning”, lifecycle “raised” and acknowledgment status “no” for the tenant TSW_Tenant0. apic1# show faults code F110473 last-days 5 severity warning lc raised ack no tenant TSW_Tenant0 Code : F110473 Severity : warning Last Transition : 2015-11-03T01:19:04.913+00:00 Lifecycle : raised DN : uni/tn-TSW_Tenant0/BD-tsw0ctx0BD1/fault-F110473 Description : TCA: ingress drop bytes rate(l2IngrBytesAg15min:dropRate) value 160462 raised above threshold 100000 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 383 Configuring Monitoring Using the show events Command Using the show events Command The show events command can combine several data filters and an entity filter to deliver a specific set of events. The command syntax is: show events [filter1 [filter2... ]] [entity-filter] Entity filters restrict the output to events of a controller, leaf, spine, or tenant. The available entity filters are listed in Entity Filters for Show Commands, on page 387. Data filters are provided to make the task of querying events easier for the user. The available data filters are: Filter Description cause fault-value cause code event-code event code controller controller information detail detailed events information end-time YYYY-MM-DDTHR-MM:SS event activity up to this time id event-id event ID last-days days event activity in the last N days last-hours hours event activity in the last N hours last-minutes minutes event activity in the last N minutes leaf [leaf-id] leaf switch information spine [spine-id] spine switch information start-time YYYY-MM-DDTHR-MM:SS event activity starting from this time tenant [name] tenant information Examples This example shows all events on leaf 101. apic1# show events leaf 101 Severity Affected Object Code ID Cause Description : : : : : : info topology/pod-1/node-101/sys/phys-[eth1/28] E4208843 8589934758 transition PhysIf eth1/28 modified Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 384 Configuring Monitoring Using the show health Command Creation Time : 2015-11-03T01:11:16.763+00:00 Using the show health Command The show health command can combine several data filters and an entity filter to deliver a specific health report. The command syntax is: show health [filter1 [filter2... ]] [entity-filter] Entity filters restrict the output to health scores of a controller, leaf, spine, or tenant. The available entity filters are listed in Entity Filters for Show Commands, on page 387. Data filters are provided to make the task of querying health easier for the user. The available data filters are: Filter Description end-time YYYY-MM-DDTHR-MM:SS health activity up to this time history historical information max-change percentage minimum change in health score percentage min-hs score maximum health score start-time YYYY-MM-DDTHR-MM:SS health activity starting from this time Examples This example shows a brief health report for all tenants. apic1# show health tenant Tenant Score Change(%) Created -----------------------------------------------------------------------infra 100 0 2015-05-12 18:45:47PDT common 100 0 2015-05-12 18:45:47PDT TSW_Tenant0 98 0 2015-05-12 18:20:58PDT mgmt 100 0 2015-05-12 18:45:47PDT This example shows all historical health records from the 4th of November that have a maximum health score of 75 that have had a minimum change of 10% for the tenant TSW_Tenant0. apic1# show health max-hs 75 min-change 10 start-time 2015-11-04T01:55:48 history tenant TSW_Tenant0 Using the show audits Command The show audits command can be used to view the audit-logs as well as the session logs for an entity. The command can combine several data filters and an entity filter to deliver a specific set of audit logs. The command syntax is: show audits [filter1 [filter2... ]] [entity-filter] Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 385 Configuring Monitoring Using the show audits Command Entity filters restrict the output to logs of a controller, leaf, spine, or tenant. The available entity filters are listed in Entity Filters for Show Commands, on page 387. Data filters are provided to make the task of querying audit logs easier for the user. The available data filters are: Filter Description action {creation | deletion | failure | modification | special | state-transition} object action indicator controller controller information detail detailed log information end-time YYYY-MM-DDTHR-MM:SS log activity up to this time id log-id log ID last-days days log activity in the last N days last-hours hours log activity in the last N hours last-minutes minutes log activity in the last N minutes leaf [leaf-id] leaf switch information spine [spine-id] spine switch information start-time YYYY-MM-DDTHR-MM:SS log activity starting from this time tenant [name] tenant information user user-name name of user Examples This example shows all audit logs in the last 45 minutes for the tenant TSW_Tenant0. apic1# show audits last-minutes 45 tenant TSW_Tenant0 Creation Time : 2015-11-03T01:11:05.708+00:00 ID : 12884902085 User : admin Action : creation Affected Object : uni/tn-TSW_Tenant0/out-T0-sub-L3OUT-1/instPl3extInstP-1/extsubnet-[192.5.1.0/24] Description : Subnet 192.5.1.0/24 created Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 386 Configuring Monitoring Using the show stats Command Using the show stats Command The show stats command can combine data filters and an entity filter to deliver a specific set of statistics. The command syntax is: show stats granularity granularity [cumulative] [history] [entity-filter] Entity filters restrict the output to statistics of a leaf, spine, or tenant. The available entity filters are listed in Entity Filters for Show Commands, on page 387. Data filters are provided to make the task of querying statistics easier for the user. The available data filters are: Filter Description cumulative cumulative statistics information granularity {5min | 15min | 1h | 1d | 1w | 1mo | 1qtr the sampling interval size which can be 5 minutes, 15 minutes, 1 hour, 1 day, 1 week, 1 month, 1 quarter, | 1year} or 1 year history historical statistics information Examples This example shows 15 minute granularity statistics for the tenant TSW_Tenant0. apic1# show stats granularity 15min tenant TSW_Tenant0 This example shows 15 minute granularity statistics for a specific port. apic1# show stats granularity 15min leaf 101 interface ethernet 1/1 Entity Filters for Show Commands Entity filters can extend many show commands to restrict the output to faults of a controller, leaf, spine, or tenant. The available entity filters are: Filter controller leaf node-id [fex] leaf node-id interface [ethernet slot/port | l3instance [ instance-name ] | mgmt [mgmt0] | portchannel | tunnel [ tunnel-name ]] leaf node-id inventory {chassis [ number ] | fans [ number ] | module [ number ] | powersupply [ number ] | supervisor [ number ]} Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 387 Configuring Monitoring Configuring SNMP Filter leaf node-id protocol {arp | bgp | coop | ipv4 | ipv6 | isis | lldp | ospf | ospfv3} leaf node-id vpc { leaf node-id vrf [ vrf-name ] spine node-id spine node-id interface [ethernet slot/port | l3instance [ instance-name ] | mgmt [mgmt0] | tunnel [ tunnel-name ]] spine node-id inventory {chassis [ number ] | fabric [ number ] | fans [ number ] | module [ number ] | powersupply [ number ] | supervisor [ number ] | system [ number ]} spine node-id protocol {arp | bgp | coop | ipv4 | ipv6 | isis | lldp | ospf | ospfv3} spine node-id vrf [ vrf-name ] tenant tenant-name tenant tenant-name application [ app-name ] [epg] tenant tenant-name bridge-domain [ bd-name ] tenant tenant-name interface bridge-domain [ bd-name ] Configuring SNMP Before You Begin To allow SNMP communications, you must configure an out-of-band contract allowing SNMP traffic, which is normally on UDP:161. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 template snmp-fabricsnmp-fabric-template-name Example: apic1(config)# template snmp-fabric Pol1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 388 Enters template snmp-fabric mode. Configuring Monitoring Configuring SNMP Step 3 Command or Action Purpose [no] snmp-server protocol enable Enables (or disables) SNMP protocol support. Example: apic1(config-template-snmp-fabric)# snmp-server protocol enable Step 4 [no] snmp-server community community-name The community is required for SNMPv2 only. Example: apic1(config-template-snmp-fabric)# snmp-server community mysecret Step 5 snmp-server contact contact-name . Example: apic1(config-template-snmp-fabric)# snmp-server contact admin80 Step 6 snmp-server location location-name Sets the location for the SNMP server. Example: apic1(config-template-snmp-fabric)# snmp-server location SanJose Step 7 Returns to global configuration mode exit Example: apic1(config-template-snmp-fabric)# exit Step 8 template pod-group pod-group-template-name Configures a pod-group template (policy). Example: apic1(config)# template pod-group allPods Step 9 inherit snmp-fabric snmp-fabric-template-name Associates the pod-profile with the previously configured pod group. Example: apic1(config-pod-group)# inherit snmp-fabric Pol1 Examples The following example configures an out-of-band contract allowing SNMP traffic in the fabric. apic1# configure apic1(config)# template snmp-fabric Pol1 apic1(config-template-snmp-fabric)# snmp-server protocol enable apic1(config-template-snmp-fabric)# snmp-server community mysecret apic1(config-template-snmp-fabric)# snmp-server contact admin80 apic1(config-template-snmp-fabric)# snmp-server location SanJose apic1(config-template-snmp-fabric)# exit apic1(config)# template pod-group allPods apic1(config-pod-group)# inherit snmp-fabric Pol1 apic1(config-pod-group)# exit apic1(config)# Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 389 Configuring Monitoring Configuring SNMP Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 390 CHAPTER 17 Configuring SPAN • Configuring SPAN and ERSPAN, page 391 Configuring SPAN and ERSPAN In the ACI Fabric, SPAN feature can be configured in three categories: • Access – for monitoring traffic originating from access ports in leaf nodes • Fabric – for monitoring traffic from fabric ports in leaf or spine nodes • Tenant – for monitoring traffic from endpoint groups (EPGs) within a tenant The following table shows the different configuration elements for each session. Session Type Sources Filters Access Local Access Ports, Port-channels local to one EPG leaf Port local to same leaf as sources Access ERSPAN Access Ports, Port-channels, VPCs among one or more leaf nodes EPG anywhere in the fabric Fabric ERSPAN Fabric ports in one or mode leaf or spine BD or VRF nodes EPG anywhere in the fabric Tenant ERSPAN EPG anywhere in the fabric EPG anywhere in the fabric EPG - Destination Configuring Local SPAN in Access Mode This is the traditional SPAN configuration local to an Access leaf node. Traffic originating from one or more access ports or port-channels can be monitored and sent to a destination port local to the same leaf node. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 391 Configuring SPAN Configuring Local SPAN in Access Mode Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] monitor access session session-name Creates an access monitoring session configuration. Example: apic1(config)# monitor access session mySession Step 3 [no] description text Example: Adds a description for this access monitoring session. If the text includes spaces, it must be enclosed in single quotes. apic1(config-monitor-access)# description "This is my SPAN session" Step 4 [no] destination interface ethernet slot/port leaf Specifies the destination interface. The destination interface cannot be a FEX port or node-id port-channel. Example: apic1(config-monitor-access)# destination interface eth 1/2 leaf 101 Step 5 [no] source interface ethernet {[ fex/ ]slot/port Specifies the source interface port or port range. | port-range} leaf node-id Example: apic1(config-monitor-access)# source interface eth 1/2 leaf 101 Step 6 [no] direction {rx | tx | both} Example: Specifies direction of traffic to be monitored. The direction can be configured independently for each source port range. apic1(config-monitor-access-source)# direction tx Step 7 [no] filter tenant tenant-name application application-name epg epg-name Filters traffic to be monitored. The filter can be configured independently for each source port range. Example: apic1(config-monitor-access-source)# filter tenant t1 application app1 epg epg1 Step 8 exit Returns to access monitor session configuration mode. Example: apic1(config-monitor-access-source)# exit Step 9 [no] source interface port-channel port-channel-name-list leaf node-id [fex fex-id] Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 392 Specifies the source interface port channel. (Enters the traffic direction and filter configuration, not shown here.) Configuring SPAN Configuring ERSPAN in Access Mode Command or Action Purpose Example: apic1(config-monitor-access)# source interface port-channel pc5 leaf 101 Step 10 Disables (or enables) the monitoring session. [no] shutdown Example: apic1(config-monitor-access)# no shut Examples This example shows how to configure a local access monitoring session. apic1# configure terminal apic1(config)# monitor access session mySession apic1(config-monitor-access)# description "This is my SPAN session" apic1(config-monitor-access)# destination interface eth 1/2 leaf 101 apic1(config-monitor-access)# source interface eth 1/1 leaf 101 apic1(config-monitor-access-source)# direction tx apic1(config-monitor-access-source)# filter tenant t1 application app1 epg epg1 apic1(config-monitor-access-source)# exit apic1(config-monitor-access)# no shut apic1(config-monitor-access)# show run # Command: show running-config monitor access session mySession # Time: Fri Nov 6 23:55:35 2015 monitor access session mySession description "This is my SPAN session" destination interface eth 1/2 leaf 101 source interface eth 1/1 leaf 101 direction tx filter tenant t1 application app1 epg epg exit exit Configuring ERSPAN in Access Mode In the ACI fabric, an access mode ERSPAN configuration can be used for monitoring traffic originating from access ports, port-channels, and vPCs in one or more leaf nodes. For an ERSPAN session, the destination is always an endpoint group (EPG) which can be deployed anywhere in the fabric. The monitored traffic is forwarded to the destination wherever the EPG is moved. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 393 Configuring SPAN Configuring ERSPAN in Access Mode Step 2 Command or Action Purpose [no] monitor access session session-name Creates an access monitoring session configuration. Example: apic1(config)# monitor access session mySession Step 3 [no] description text Example: Adds a description for this monitoring session. If the text includes spaces, it must be enclosed in single quotes. apic1(config-monitor-access)# description "This is my access ERSPAN session" Step 4 [no] destination tenant tenant-name application Specifies the destination interface as a tenant application-name epg epg-name destination-ip and enters destination configuration mode. dest-ip-address source-ip-prefix src-ip-address Example: apic1(config-monitor-access)# destination tenant t1 application app1 epg epg1 destination-ip 192.0.20.123 source-ip-prefix 10.0.20.1 Step 5 [no] erspan-id flow-id Example: Configures the ERSPAN ID for the ERSPAN session. The ERSPAN range is from 1 to 1023. apic1(config-monitor-access-dest)# erspan-id 100 Step 6 [no] ip dscp dscp-code Example: Configures the differentiated services code point (DSCP) value of the packets in the ERSPAN traffic. The range is from 0 to 64. apic1(config-monitor-access-dest)# ip dscp 42 Step 7 [no] ip ttl ttl-value Example: Configures the IP time-to-live (TTL) value for the ERSPAN traffic. The range is from 1 to 255. apic1(config-monitor-access-dest)# ip ttl 16 Step 8 [no] mtu mtu-value Example: Configures the maximum transmit unit (MTU) size for the ERSPAN session. The range is 64 to 9216 bytes. apic1(config-monitor-access-dest)# mtu 9216 Step 9 exit Returns to monitor access configuration mode. Example: apic1(config-monitor-access-dest)# Step 10 [no] source interface ethernet {[ fex/ ]slot/port | Specifies the source interface port or port range. port-range} leaf node-id Example: apic1(config-monitor-access)# source interface eth 1/2 leaf 101 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 394 Configuring SPAN Configuring ERSPAN in Access Mode Step 11 Command or Action Purpose [no] source interface port-channel port-channel-name-list leaf node-id [fex fex-id] Specifies the source interface port-channel. Example: apic1(config-monitor-access)# source interface port-channel pc1 leaf 101 Step 12 [no] source interface vpc vpc-name-list leaf node-id1 node-id2 [fex fex-id1 fex-id2] Specifies the source interface vPC. Example: apic1(config-monitor-access)# source interface vpc pc1 leaf 101 102 Step 13 [no] direction {rx | tx | both} Specifies direction of traffic to be monitored. The direction can be configured independently for each source port range. Example: apic1(config-monitor-access-source)# direction tx Step 14 [no] filter tenant tenant-name application application-name epg epg-name Filters traffic to be monitored. The filter can be configured independently for each source port range. Example: apic1(config-monitor-access-source)# filter tenant t1 application app1 epg epg1 Step 15 Returns to access monitor session configuration mode. exit Example: apic1(config-monitor-access-source)# exit Step 16 Disables (or enables) the monitoring session. [no] shutdown Example: apic1(config-monitor-access)# no shut Examples This example shows how to configure an ERSPAN access monitoring session. apic1# configure terminal apic1(config)# monitor access session mySession apic1(config-monitor-access)# description "This is apic1(config-monitor-access)# destination tenant t1 192.0.20.123 source-ip-prefix 10.0.20.1 apic1(config-monitor-access-dest)# erspan-id 100 apic1(config-monitor-access-dest)# ip dscp 42 apic1(config-monitor-access-dest)# ip ttl 16 apic1(config-monitor-access-dest)# mtu 9216 apic1(config-monitor-access-dest)# exit apic1(config-monitor-access)# source interface eth apic1(config-monitor-access-source)# direction tx apic1(config-monitor-access-source)# filter tenant apic1(config-monitor-access-source)# exit my access ERSPAN session" application app1 epg epg1 destination-ip 1/1 leaf 101 t1 application app1 epg epg1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 395 Configuring SPAN Configuring ERSPAN in Fabric Mode apic1(config-monitor-access)# no shut apic1(config-monitor-access)# show run # Command: show running-config monitor access session mySession # Time: Fri Nov 6 23:55:35 2015 monitor access session mySession description "This is my ERSPAN session" source interface eth 1/1 leaf 101 direction tx filter tenant t1 application app1 epg epg1 exit destination tenant t1 application app1 epg epg1 destination-ip 192.0.20.123 source-ip-prefix 10.0.20.1 ip dscp 42 ip ttl 16 erspan-id 9216 mtu 9216 exit exit This example shows how to configure a port-channel as a monitoring source. apic1(config-monitor-access)# source interface port-channel pc3 leaf 105 This example shows how to configure a one leg of a vPC as a monitoring source. apic1(config-monitor-access)# source interface port-channel vpc3 leaf 105 This example shows how to configure a range of ports from FEX 101 as a monitoring source. apic1(config-monitor-access)# source interface eth 101/1/1-2 leaf 105 Configuring ERSPAN in Fabric Mode In the ACI fabric, a fabric mode ERSPAN configuration can be used for monitoring traffic originating from one or more fabric ports in leaf or spine nodes. Local SPAN is not supported in fabric mode. For an ERSPAN session, the destination is always an endpoint group (EPG) which can be deployed anywhere in the fabric. The monitored traffic is forwarded to the destination wherever the EPG is moved. In the fabric mode, only fabric ports are allowed as source, but both leaf and spine switches are allowed. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] monitor fabric session session-name Example: apic1(config)# monitor fabric session mySession Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 396 Creates a fabric monitoring session configuration. Configuring SPAN Configuring ERSPAN in Fabric Mode Step 3 Command or Action Purpose [no] description text Adds a description for this monitoring session. If the text includes spaces, it must be enclosed in single quotes. Example: apic1(config-monitor-fabric)# description "This is my fabric ERSPAN session" Step 4 [no] destination tenant tenant-name application Specifies the destination interface as a tenant application-name epg epg-name destination-ip and enters destination configuration mode. dest-ip-address source-ip-prefix src-ip-address Example: apic1(config-monitor-fabric)# destination tenant t1 application app1 epg epg1 destination-ip 192.0.20.123 source-ip-prefix 10.0.20.1 Step 5 [no] erspan-id flow-id Configures the ERSPAN ID for the ERSPAN session. The ERSPAN range is from 1 to 1023. Example: apic1(config-monitor-fabric-dest)# erspan-id 100 Step 6 [no] ip dscp dscp-code Configures the differentiated services code point (DSCP) value of the packets in the ERSPAN traffic. The range is from 0 to 64. Example: apic1(config-monitor-fabric-dest)# ip dscp 42 Step 7 [no] ip ttl ttl-value Configures the IP time-to-live (TTL) value for the ERSPAN traffic. The range is from 1 to 255. Example: apic1(config-monitor-fabric-dest)# ip ttl 16 Step 8 [no] mtu mtu-value Configures the maximum transmit unit (MTU) size for the ERSPAN session. The range is 64 to 9216 bytes. Example: apic1(config-monitor-fabric-dest)# mtu 9216 Step 9 Returns to monitor access configuration mode. exit Example: apic1(config-monitor-fabric-dest)# Step 10 [no] source interface ethernet {slot/port | port-range} switch node-id Specifies the source interface port or port range. Example: apic1(config-monitor-fabric)# source interface eth 1/2 switch 101 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 397 Configuring SPAN Configuring ERSPAN in Tenant Mode Step 11 Command or Action Purpose [no] direction {rx | tx | both} Specifies direction of traffic to be monitored. The direction can be configured independently for each source port range. Example: apic1(config-monitor-fabric-source)# direction tx Step 12 [no] filter tenant tenant-name bd bd-name Filters traffic by bridge domain. Example: apic1(config-monitor-fabric-source)# filter tenant t1 bd bd1 Step 13 [no] filter tenant tenant-name vrf vrf-name Filters traffic by VRF. Example: apic1(config-monitor-fabric-source)# filter tenant t1 vrf vrf1 Step 14 Returns to access monitor session configuration mode. exit Example: apic1(config-monitor-fabric-source)# exit Step 15 [no] shutdown Disables (or enables) the monitoring session. Example: apic1(config-monitor-fabric)# no shut Examples This example shows how to configure an ERSPAN fabric monitoring session. apic1# configure terminal apic1(config)# monitor access session mySession apic1(config-monitor-fabric)# description "This is apic1(config-monitor-fabric)# destination tenant t1 192.0.20.123 source-ip-prefix 10.0.20.1 apic1(config-monitor-fabric-dest)# erspan-id 100 apic1(config-monitor-fabric-dest)# ip dscp 42 apic1(config-monitor-fabric-dest)# ip ttl 16 apic1(config-monitor-fabric-dest)# mtu 9216 apic1(config-monitor-fabric-dest)# exit apic1(config-monitor-fabric)# source interface eth apic1(config-monitor-fabric-source)# direction tx apic1(config-monitor-fabric-source)# filter tenant apic1(config-monitor-fabric-source)# filter tenant apic1(config-monitor-fabric-source)# exit apic1(config-monitor-fabric)# no shut my fabric ERSPAN session" application app1 epg epg1 destination-ip 1/1 switch 101 t1 bd bd1 t1 vrf vrf1 Configuring ERSPAN in Tenant Mode In the ACI fabric, a tenant mode ERSPAN configuration can be used for monitoring traffic originating from endpoint groups within a tenant. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 398 Configuring SPAN Configuring ERSPAN in Tenant Mode In the tenant mode, traffic originating from a source EPG is sent to a destination EPG within the same tenant. The monitoring of traffic is not impacted if the source or destination EPG is moved within the fabric. Procedure Step 1 Command or Action Purpose configure Enters global configuration mode. Example: apic1# configure Step 2 [no] monitor tenant tenant-name session session-name Creates a tenant monitoring session configuration. Example: apic1(config)# monitor tenant session mySession Step 3 [no] description text Adds a description for this access monitoring session. If the text includes spaces, it must be enclosed in single quotes. Example: apic1(config-monitor-tenant)# description "This is my tenant ERSPAN session" Step 4 Specifies the destination interface as a tenant [no] destination tenant tenant-name application application-name epg epg-name and enters destination configuration mode. destination-ip dest-ip-address source-ip-prefix src-ip-address Example: apic1(config-monitor-tenant)# destination tenant t1 application app1 epg epg1 destination-ip 192.0.20.123 source-ip-prefix 10.0.20.1 Step 5 [no] erspan-id flow-id Configures the ERSPAN ID for the ERSPAN session. The ERSPAN range is from 1 to 1023. Example: apic1(config-monitor-tenant-dest)# erspan-id 100 Step 6 [no] ip dscp dscp-code Configures the differentiated services code point (DSCP) value of the packets in the ERSPAN traffic. The range is from 0 to 64. Example: apic1(config-monitor-tenant-dest)# ip dscp 42 Step 7 [no] ip ttl ttl-value Configures the IP time-to-live (TTL) value for the ERSPAN traffic. The range is from 1 to 255. Example: apic1(config-monitor-tenant-dest)# ip ttl 16 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 399 Configuring SPAN Configuring ERSPAN in Tenant Mode Step 8 Command or Action Purpose [no] mtu mtu-value Configures the maximum transmit unit (MTU) size for the ERSPAN session. The range is 64 to 9216 bytes. Example: apic1(config-monitor-tenant-dest)# mtu 9216 Step 9 exit Returns to monitor access configuration mode. Example: apic1(config-monitor-tenant-dest)# Step 10 [no] source application application-name epg Specifies the source interface port or port range. epg-name Example: apic1(config-monitor-tenant)# source application app2 epg epg5 Step 11 [no] direction {rx | tx | both} Example: Specifies direction of traffic to be monitored. The direction can be configured independently for each source port range. apic1(config-monitor-tenant-source)# direction tx Step 12 exit Returns to access monitor session configuration mode. Example: apic1(config-monitor-tenant-source)# exit Step 13 [no] shutdown Disables (or enables) the monitoring session. Example: apic1(config-monitor-tenant)# no shut Examples This example shows how to configure an ERSPAN tenant monitoring session. apic1# configure terminal apic1(config)# monitor access session mySession apic1(config-monitor-tenant)# description "This is my tenant ERSPAN session" apic1(config-monitor-tenant)# destination tenant t1 application app1 epg epg1 destination-ip 192.0.20.123 source-ip-prefix 10.0.20.1 apic1(config-monitor-tenant-dest)# erspan-id 100 apic1(config-monitor-tenant-dest)# ip dscp 42 apic1(config-monitor-tenant-dest)# ip ttl 16 apic1(config-monitor-tenant-dest)# mtu 9216 apic1(config-monitor-tenant-dest)# exit apic1(config-monitor-tenant)# source application app2 epg epg5 apic1(config-monitor-tenant-source)# direction tx apic1(config-monitor-tenant-source)# exit apic1(config-monitor-tenant)# no shut Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 400 CHAPTER 18 Applying the show running config Output to Another Cisco APIC This section explains how to use the export config output on another Cisco APIC. config and import config CLIs to use the show running • About Import and Export Configurations, page 401 • Import and Export Configuration Guidelines and Limitations, page 401 • Exporting a CLI Configuration, page 402 • Importing a CLI Configuration, page 402 About Import and Export Configurations The import config and export config commands enable you to apply the show running config output to another Cisco APIC. This section contains the guidelines for these commands and demonstrates how the commands are executed. Import and Export Configuration Guidelines and Limitations This section explains the guidelines and limitations for the export config and import config commands. • Passwords and other encrypted data are not included in the configuration file. • Some REST API configurations may not be compatible with CLI configurations; this may cause errors when applying a configuration file to a Cisco APIC. • Some features require configurations to be in a specific order. These configurations are validated when performed through the CLI. Configurations through the REST API, however, are not validated and may cause errors when running the imported file due to missing configurations. • Interactive commands are prefixed with a "#" and ignored when running the configuration file. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 401 Applying the show running config Output to Another Cisco APIC Exporting a CLI Configuration Exporting a CLI Configuration This procedure shows how to export a configuration to a text file. Procedure Step 1 Command or Action Purpose configure Enters configuration mode. Example: dev4-ifc1# configure Step 2 leaf ID Identifies the leaf with the configuration to be exported. Example: dev4-ifc1(config)# Step 3 leaf 101 interface ethernet slot/port Identifies the slot number and port number for an existing Ethernet interface. Example: dev4-ifc1(config-leaf)# interface ethernet 1/34 Step 4 export-config result-file-name Exports the configuration to a specified file name. Example: dev4-ifc1(config-leaf-if)# export-config /tmp/showRunnLeaf101.txt Example This example shows how to configure export-config. dev4-ifc1# config dev4-ifc1(config)# leaf 101 dev4-ifc1(config-leaf)# interface ethernet 1/34 dev4-ifc1(config-leaf-if)# export-config /tmp/showRunnLeaf101.txt dev4-ifc1(config-leaf-if)# cat /tmp/showRunnLeaf101.txt config # Command: show running-config leaf 101 interface ethernet 1 / 34 # Time: Fri Sep 23 16:03:48 2016 leaf 101 interface ethernet 1/34 switchport trunk allowed vlan 602 tenant t1 external-svi l3out l3ext1sub1 exit exit dev4-ifc1(config-leaf-if)# Importing a CLI Configuration This procedure shows how to import a configuration from a text file. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 402 Applying the show running config Output to Another Cisco APIC Importing a CLI Configuration Procedure Command or Action Step 1 Purpose import-config file-name Example: dev4-ifc1(config-tenant)# import-config /tmp/showRunnLeaf101.txt config # Command: show running-config leaf 101 interface ethernet 1 / 34 # Time: Fri Sep 23 16:03:48 2016 leaf 101 interface ethernet 1/34 switchport trunk allowed vlan 602 tenant t1 external-svi l3out l3ext1sub1 exit exit dev4-ifc1(config)# Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 403 Applying the show running config Output to Another Cisco APIC Importing a CLI Configuration Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 404 CHAPTER 19 Configuring a Forwarding Scale Profile Policy • Overview, page 405 • Supported Platforms for the IPv4 Forwarding Scale Profile Policy, page 406 • Configuring the Forwarding Scale Profile Policy Using the NX-OS-Style CLI, page 406 Overview The forwarding scale profile policy enables you to choose between Dual Stack (the default profile) and IPv4 Scale. A forwarding scale profile policy that is set to Dual Stack provides scalability of up to 6K endpoints for IPv6 configurations and up to 12K endpoints for IPv4 configurations. The IPv4 Scale option enables systems with no IPv6 configurations to increase scalability with up to 24K IPv4 endpoints. Note • Because the IPv4 forwarding scale profile policy does not support IPv6 configurations, all IPv6 configurations should be removed from switches configured with the IPv4 forwarding scale profile policy. • Applying the IPv4 forwarding scale profile policy to supported switches will cause a reload that is triggered with at least a 60-second delay from the time of the configuration. The switches will come back up with the newly applied policy. Any unsupported switches will be ignored. For a list of supported switches, see Supported Platforms for the IPv4 Forwarding Scale Profile Policy, on page 406 • vPCs associated with different scale profile settings are not supported. The vPC members must be configured with the same scale profile settings. • When performing an upgrade and clean reload on switches with an IPv4 Scale configuration, the switch will first boot up in default mode and reload a second time after getting the IPv4 Scale configuration from the APIC. A stateful reload works the same as it does in default mode. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 405 Configuring a Forwarding Scale Profile Policy Supported Platforms for the IPv4 Forwarding Scale Profile Policy Supported Platforms for the IPv4 Forwarding Scale Profile Policy The IPv4 forwarding scale profile policy is only supported on the following switches: • Cisco Nexus 9300-EX Series switches • N9K-C93180YC-FX • N9K-C9348GC-FXP • N9K-C9348GC-FXP • N9K-C93108TC-FX • N9K-C93108YC-FX Configuring the Forwarding Scale Profile Policy Using the NX-OS-Style CLI Before You Begin The forwarding scale profile policy can be set to dual stack (default) or IPv4 scale. The IPv4 forwarding scale profile policy requires supported switches. For a list of supported switches, see Supported Platforms for the IPv4 Forwarding Scale Profile Policy, on page 406. Note The switches that support the IPv4 forwarding scale profile policy will reload after the profile policy is applied. Switches that do not support the IPv4 forwarding scale profile policy will be ignored. This section demonstrates how to configure the forwarding scale profile policy using the NX-OS-style CLI. Procedure Step 1 Command or Action Purpose configure Enter global configuration mode. Example: apic1# configure Step 2 no scale-profile name Define the scale-profile policy. Example: apic1(config)# scale-profile testFwdScaleProf Step 3 profile-type {dual-stack | ipv4 } Example: apic1(config-scale-profile)# profile-type ipv4 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 406 Set the profile type. Configuring a Forwarding Scale Profile Policy Configuring the Forwarding Scale Profile Policy Using the NX-OS-Style CLI Step 4 Command or Action Purpose exit Returns back to global configuration. Example: apic1(config-scale-profile)# exit Step 5 template leaf-policy-group leaf_group_name Define the leaf policy group. Example: apic1(config)# template leaf-policy-group samplePolicyGrp Step 6 Step 7 scale-profile name Example: Configures the relation between the scale-profile policy and the leaf policy group. apic1(config-leaf-policy-group)# scale-profile testFwdScaleProf Note exit Returns back to global configuration. When applying the ipv4 profile type to a leaf policy group, supported switches will reload. Switches that do not support the IPv4 scale profile will be ignored. For a list of switches that support the IPv4 scale profile policy, see Supported Platforms for the IPv4 Forwarding Scale Profile Policy, on page 406. Example: apic1(config-leaf-policy-group)# exit Step 8 leaf-profile leaf_profile_name Configures a leaf profile. Example: apic1(config)# leaf-profile sampleLeafProf Step 9 leaf-group leaf_group_name Specifies a group of leaf switches. Example: apic1(config-leaf-profile)# leaf-group sampleLeafGrp Step 10 leaf leaf_group_number Adds leaf switches to the leaf group. Example: apic1(config-leaf-profile)# leaf 201 Step 11 leaf-policy-group leaf_policy_group_name Specifies the leaf policy group to be associated to the leaf switches. Example: apic1(config-leaf-group)# leaf-policy-group samplePolicyGrp Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 407 Configuring a Forwarding Scale Profile Policy Configuring the Forwarding Scale Profile Policy Using the NX-OS-Style CLI Step 12 Command or Action Purpose exit Exit command mode. Example: apic1(config-leaf-policy-group)# exit Step 13 [show] running-config Display the current running configuration. Example: apic1(config)# show running-config # Command: show running-config scale-profile testFwdScaleProf # Time: Thu Jul 27 22:31:29 2017 scale-profile testFwdScaleProf profile-type ipv4 exit apic1(config-scale-profile)# Step 14 [show] template leaf-policy-group Display the current running configuration. Example: Examples This example shows how to configure a the IPv4 scale profile policy. apic1# configure apic1(config)# scale-profile testFwdScaleProf apic1(config-scale-profile)# profile-type ipv4 apic1(config-scale-profile)# exit apic1(config)# template leaf-policy-group samplePolicyGrp apic1(config-leaf-policy-group)# scale-profile testFwdScaleProf apic1(config-leaf-policy-group)# exit apic1(config)# leaf-profile sampleLeafProf apic1(config-leaf-profile)# leaf-group sampleLeafGrp apic1(config-leaf-profile)# leaf 201 apic1(config-leaf-group)# leaf-policy-group samplePolicyGrp apic1(config-leaf-group)# show running-config scale-profile testFwdScaleProf # Command: show running-config scale-profile testFwdScaleProf # Time: Thu Jul 27 22:31:29 2017 scale-profile testFwdScaleProf profile-type ipv4 exit apic1(config-leaf-group)# show running-config template leaf-policy-group samplePolicyGrp # Command: show running-config template leaf-policy-group samplePolicyGrp # Time: Tue Aug 1 11:19:44 2017 template leaf-policy-group samplePolicyGrp scale-profile testFwdScaleProf exit apic1(config-leaf-group)# show running-config leaf-profile sampleLeafProf # Command: show running-config leaf-profile sampleLeafProf # Time: Tue Aug 1 11:19:58 2017 leaf-profile sampleLeafProf leaf-group sampleLeafGrp leaf 201 leaf-policy-group samplePolicyGrp exit Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 408 APPENDIX A Verified Scalability Using the CLI • CLI Scalability Limits, page 409 CLI Scalability Limits Configurable Option Scale Number of tenants 500 Number of Layer 3 (L3) contexts 300 Number of endpoint groups (EPGs) 3,500 Number of endpoints (EPs) 20,000 Number of bridge domains (BDs) 3,500 Number of BGP + number of OSPF sessions + EIGRP 300 (for external connection) Maximum number of vPCs 48 Maximum number of PCs, access ports 48 Maximum number of encaps per access port 1,750 Number of multicast groups 8,000 Maximum number of vzAny provided contracts 16 Maximum number of vzAny consumed contracts 16 Maximum amount of encaps per endpoint group 2 static, 1 dynamic Security TCAM size 4,000 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 409 Verified Scalability Using the CLI CLI Scalability Limits Configurable Option Scale Number of VRFs 500 Separate-Config-Set Tenants 100 Endpoint groups 1,000 Bridge domains 500 VRFs 100 SPAN destinations 3 NTP servers 2 Contracts 100 DNS servers 2 Syslog servers 1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 410 APPENDIX B Use Case: Three-Tier Application with Transit Topology • About Deploying a Three-Tier Application with Transit Topology, page 411 • Deploying a Three-Tier Application, page 413 • Transit Routing with OSPF and BGP, page 415 About Deploying a Three-Tier Application with Transit Topology Typically, the APIC fabric hosts a three-tier application within a tenant network. In this example, the application is implemented by using three servers (a web server, an application server, and a database server). See the following figure for an example of a three-tier application. Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 411 Use Case: Three-Tier Application with Transit Topology About Deploying a Three-Tier Application with Transit Topology The web server has the HTTP filter, the application server has the Remote Method Invocation (RMI) filter, and the database server has the Structured Query Language (SQL) filter. The application server consumes the SQL contract to communicate with the database server. The web server consumes the RMI contract to communicate with the application server. The traffic enters from the web server and communicates with the application server. The application server then communicates with the database server, and the traffic can also communicate externally. To deploy the three-tier application, you must create the required EPGs, filters, and contracts. A filter specifies the data protocols to be allowed or denied by a contract that contains the filter. A contract can contain multiple subjects. A subject can be used to realize uni- or bidirectional filters. A unidirectional filter is a filter that is used in one direction, either from consumer-to-provider (IN) or from provider-to-consumer (OUT) filter. A bidirectional filter is the same filter that is used in both directions. It is not reflexive. Contracts are policies that enable inter-End Point Group (inter-EPG) communication. These policies are the rules that specify communication between application tiers. If no contract is attached to the EPG, inter-EPG communication is disabled by default. No contract is required for intra-EPG communication because intra-EPG communication is always allowed. About Transit Routing Transit routing enables border routers to perform bidirectional redistribution with other routing domains. Bidirectional redistribution passes routing information from one routing domain to another. Such redistribution Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 412 Use Case: Three-Tier Application with Transit Topology Deploying a Three-Tier Application lets the ACI fabric provide full IP connectivity between different routing domains. Doing so can also provide redundant connectivity by enabling backup paths between routing domains. For more information, see "ACI Transit Routing" in the Cisco ACI Fundamentals Guide. Deploying a Three-Tier Application Configure the tenant VRF and bridge domain. apic1(config)# tenant t1 apic1(config-tenant)# vrf context v1 apic1(config-tenant-vrf)# contract enforce apic1(config-tenant)# bridge-domain b1 apic1(config-tenant-bd)# vrf member v1 apic1(config-tenant)# interface bridge-domain b1 apic1(config-tenant-interface)# ip address 159.10.10.1/24 scope public apic1(config-tenant-interface)# exit Configure three EPGs: web, app, and db. apic1(config-tenant)# application retail apic1(config-tenant-app)# epg web apic1(config-tenant-app-epg)# bridge-domain member b1 apic1(config-tenant-app-epg)# contract provider web apic1(config-tenant-app-epg)# contract consumer app apic1(config-tenant-app)# epg app apic1(config-tenant-app-epg)# bridge-domain member b1 apic1(config-tenant-app-epg)# contract provider app apic1(config-tenant-app-epg)# contract consumer db apic1(config-tenant-app)# epg db apic1(config-tenant-app-epg)# bridge-domain member b1 apic1(config-tenant-app-epg)# contract provider db Configure VLAN domain. apic1(config)# vlan-domain dom100 apic1(config-vlan)# vlan 100-200 Create port-channel and deploy the web EPG. apic1(config)# leaf 101 apic1(config-leaf)# interface ethernet 1/2-5 apic1(config-leaf-if)# channel-group po1 apic1(config-leaf)# interface port-channel po1 apic1(config-leaf-if)# vlan-domain member dom100 apic1(config-leaf-if)# switchport trunk allowed vlan 101 tenant t1 application retail epg web Create a vPC and deploy app and db EPGs. apic1(config)# leaf 101,102 apic1(config-leaf)# interface ethernet 1/6,1/7 apic1(config-leaf-if)# channel-group vpc1 vpc apic1(config)# vpc domain explicit 100 leaf 101 102 apic1(config)# vpc context leaf 101 102 apic1(config-vpc)# interface vpc vpc1 apic1(config-vpc-if)# vlan-domain member dom100 apic1(config-vpc-if)# switchport trunk allowed vlan 102 tenant t1 application retail epg app apic1(config-vpc-if)# switchport trunk allowed vlan 103 tenant t1 application retail epg Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 413 Use Case: Three-Tier Application with Transit Topology Deploying a Three-Tier Application db Configure MP-BGP. apic1(config)# bgp-fabric apic1(config-bgp-fabric)# asn 100 apic1(config-bgp-fabric)# route-reflector spine 104,105 Configure External-l3 EPG. apic1(config-tenant)# external-l3 epg l3epg1 apic1(config-tenant-l3ext-epg)# vrf member v1 apic1(config-tenant-l3ext-epg)# match ip 173.10.1.0/24 apic1(config-tenant-l3ext-epg)# contract consumer web Configure VRF on Leaf , route-map and deploy external-l3 EPG. apic1(config)# leaf 103 apic1(config-leaf)# vrf context tenant t1 vrf v1 apic1(config-leaf-vrf)# external-l3 epg l3epg1 apic1(config-leaf-vrf)# route-map map1 apic1(config-leaf-vrf-route-map)# match bridge-domain b1 Configure OSPF area on a sub-Interface. apic1(config-leaf)# router ospf default apic1(config-leaf-ospf)# vrf member tenant t1 vrf v1 apic1(config-leaf-ospf-vrf)# area 0.0.0.1 route-map map1 out apic1(config-leaf)# interface ethernet 1/2 apic1(config-leaf-if)# no switchport apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf)# interface ethernet 1/2.150 apic1(config-leaf-if)# vrf member tenant t1 vrf v1 apic1(config-leaf-if)# ip address 169.10.10.1/24 apic1(config-leaf-if)# ip router ospf default area 0.0.0.1 Configure filters. apic1(config-tenant)# access-list http apic1(config-tenant-acl)# match tcp dest 80 apic1(config-tenant-acl)# match tcp dest 443 apic1(config-tenant)# access-list rmi apic1(config-tenant-acl)# match tcp dest 1099 apic1(config-tenant)# access-list sql apic1(config-tenant-acl)# match tcp dest 1521 Configure contracts. apic1(config-tenant)# contract rmi apic1(config-tenant-contract)# subject rmi apic1(config-tenant-contract-subj)# access-group rmi both apic1(config-tenant)# contract web apic1(config-tenant-contract)# subject web apic1(config-tenant-contract-subj)# access-group http both apic1(config-tenant)# contract db apic1(config-tenant-contract)# subject sql apic1(config-tenant-contract-subj)# access-group sql both Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 414 Use Case: Three-Tier Application with Transit Topology Transit Routing with OSPF and BGP Transit Routing with OSPF and BGP This procedure configures transit routing between Site1 and Site2 for the three-tier application described in Deploying a Three-Tier Application in this chapter. Configure External-l3 EPG (l3epg2) for Site2. apic1(config-tenant)# external-l3 epg l3epg2 apic1(config-tenant-l3ext-epg)# vrf member v1 apic1(config-tenant-l3ext-epg)# match ip 174.10.1.0/24 apic1(config-tenant-l3ext-epg)# contract consumer transit apic1(config)# leaf 102 apic1(config-leaf)# vrf context tenant t1 vrf v1 apic1(config-leaf-vrf)# external-l3 epg l3epg2 Configure BGP connectivity over External SVI and export route corresponding to Site1. apic1(config)# leaf 102 apic1(config-leaf-vrf)# route-map map200 apic1(config-leaf-vrf-route-map)# ip prefix-list p1 match 173.10.1.0/24 apic1(config-leaf-vrf-route-map)# match prefix-list p1 apic1(config-leaf-vrf-route-map-match)# set community extended 200:1 replace apic1(config-leaf)# interface vlan 160 apic1(config-leaf-if)# vrf member tenant t1 vrf v1 apic1(config-leaf-if)# ip address 208.1.1.2/24 apic1(config-leaf)# interface ethernet 1/11 apic1(config-leaf-if)# vlan-domain member dom1 apic1(config-leaf-if)# switchport trunk allowed vlan 160 tenant t1 external-svi apic1(config-leaf)# router bgp 100 apic1(config-bgp)# vrf member tenant apic1(config-leaf-bgp-vrf)# neighbor apic1(config-leaf-bgp-vrf-neighbor)# apic1(config-leaf-bgp-vrf-neighbor)# apic1(config-leaf-bgp-vrf-neighbor)# t1 vrf v1 208.1.1.1 remote-as 200 update-source vlan 160 route-map map200 out Configure contract provider on l3epg1 (Site1) to establish connection with l3epg2 (Site2) apic1(config-tenant)# external-l3 epg l3epg1 apic1(config-tenant-l3ext-epg)# contract provider transit Configure a route-map on Site1 to export the route corresponding to Site2. apic1(config)# leaf 103 apic1(config-leaf-vrf)# route-map map1 apic1(config-leaf-vrf-route-map)# ip prefix-list p1 match 174.10.1.0/24 apic1(config-leaf-vrf-route-map)# match prefix-list p1 apic1(config-leaf-vrf-route-map-match)# set metric 100 Configure ACL and contract for transit routing. apic1(config)# tenant t1 apic1(config-tenant)# access-list acl1 apic1(config-tenant-acl)# match ip apic1(config-tenant)# contract transit apic1(config-tenant-contract)# subject ip apic1(config-tenant-contract-subj)# access-group acl1 both Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 415 Use Case: Three-Tier Application with Transit Topology Transit Routing with OSPF and BGP Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 416 APPENDIX C Examples: Show Commands • Examples: Show Commands, page 417 Examples: Show Commands show running-config show running-config “local” to the current mode. apic1(config)# leaf 103 apic1(config-leaf)# interface ethernet 1/2.150 apic1(config-leaf-if)# show running-config # Command: show running-config leaf 103 interface ethernet 1 / 2 . 150 # Time: Tue Dec 8 08:08:37 2015 leaf 103 interface ethernet 1/2.150 vrf member tenant t1 vrf v1 ip address 169.10.10.1/24 ip router ospf default area 0.0.0.1 exit exit show running-config with filters. apic1(config-leaf)# interface ethernet 1/2.150 apic1(config-leaf-if)# show running-config leaf 103 # Command: show running-config leaf 103 # Time: Tue Dec 8 08:10:02 2015 leaf 103 vrf context tenant t1 vrf v1 external-l3 epg l3epg1 route-map map1 ip prefix-list p1 permit 181.1.1.0/24 match bridge-domain b1 match prefix-list p1 … show vpc, port-channel show vpc map apic1(config-leaf-if)# show vpc map Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 417 Examples: Show Commands Examples: Show Commands Legends: N/D : Not Deployed Virtual Port-Channel Name Domain VPC Leaf Id, Name ------------------------- ------ --- ------------- vpc1 100 1 vpc1 100 1 Fex Id ------ PC Id Ports ----- -------------------- 101,leaf1 po2 eth1/6-7, eth1/40-41 102,leaf2 po1 eth1/6-7, eth1/40-41 show port-channel map apic1(config-leaf-if)# show port-channel map Legends: N/D : Not Deployed PC: Port Channel VPC: Virtual Port Channel Port-Channel Name ----------------po1 po1 Type ---PC PC Leaf ID, Name -----------------101,leaf1 102,leaf2 vpc1 vpc1 VPC VPC 101,leaf1 102,leaf2 Fex Id ------ Port Channel ------------po1 po2 Ports -------------------eth1/2-5, eth1/32-33 eth1/32-33 po2 po1 eth1/6-7, eth1/40-41 eth1/6-7, eth1/40-41 show vlan-domain show vlan-domain name dom100 apic1# show vlan-domain name dom100 Legend: vlanscope: L (Portlocal). Default is global vlan-domain : dom100 Type : All vlan : 100-200(static) Leaf -------101 Interface ---------PC: po1 Vlan ---101 Type --------App-Epg Usage -----------Tenant: t1 App: retail Epg: web Operational State ----------------b1: down web: down Operational Vlan ---------------b1: vlan-18 web: vlan-21 101,102 vPC: vpc1 102 App-Epg Tenant: t1 App: retail Epg: app b1: down app: down b1: vlan-18 app: vlan-19 101,102 vPC: vpc1 103 App-Epg Tenant: t1 App: retail Epg: db b1: down db: down b1: vlan-18 db: vlan-20 102 eth1/11 160 Ext-svi Tenant: t1 Vrf: v1 l2: down l3: down vlan-18 103 eth1/2 150 Ext-subIf Tenant: t1 Vrf: v1 - eth1/2.14 show tenant show tenant t1 detail apic1# show tenant t1 detail Detailed view for Tenant t1 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 418 Examples: Show Commands Examples: Show Commands Security Information: Security Domain ---------------------------------------VRF Information: VRF -------------------v1 Policy Enforcement -------------------enforced Bridge-Domain Information: BD VRF -------------------- -------------------b1 v1 Static VLAN Information: Node VLANs -------- ---------------------------------------101 101 101 102 102,103 VLAN Domains -----------------------------dom100 dom100 Static Application EPg Information: Node Interface App:AEPg BD Contract -------- ------------------------------ -------------------- --------------------------101 port-channel po1 retail:web b1 web, app 101 102 vpc vpc1 retail:db,retail:app b1 app, db Application EPg Information: App:AEPg BD -------------------- -------------------retail:app b1 retail:db b1 retail:web b1 External L2 EPg Information: external-l2 BD -------------------- -------------------External L3 EPg Information: external-l3 VRF -------------------- -------------------l3epg1 v1 l3epg2 v1 show external-l3 show external-l3 interfaces apic1# show external-l3 interfaces Node Tenant VRF Oper IP ----- ------------ -----------------102 t1 v1 Interface Oper Interface IP Address ---------------- ---------------- -------------- vlan-160 eth1/11 vlan18 208.1.1.2/24 up 169.10.10.1/24 up eth1/11 103 t1 v1 eth1/2.150 eth1/2.14 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 419 Examples: Show Commands Examples: Show Commands show external-l3 epg apic1# show external-l3 epg Name Flags State ---------- ---------------------------------t1: vxlan: 2457600 l3epg1 vrf: v1 Target dscp: unspecified qosclass: unspecified Contracts --------Provided: transit Consumed: web t1: vxlan: 2457600 disabled l3epg2 vrf: v1 disabled Target dscp: unspecified disabled qosclass: unspecified Contracts --------Provided: Consumed: transit Match Node Entry -------------- ---------- --------------- node-103 173.10.1.0/24 node-101 173.10.1.0/24 node-102 173.10.1.0/24 Oper 173.10.1.0/24 173.10.1.0/24 show external-l3 ospf apic1# show external-l3 ospf tenant t1 vrf v1 Area Id : 0.0.0.1 Tenant : t1 Vrf : v1 User Config : Node ID Area Properties ------------------------------------------------------------------------103 Type: nssa, Cost: 1, Control: redistribute,summary Configuration : Node ID ---103 Operational Router ID --------------10.1.0.103 Interfaces : Configuration : Node ID ---103 Route Map ---------------map1 Area Oper. Props ----------------------------------Type: nssa, Cost: 1, Control: redistribute,summary, AreaId: 0.0.0.1 Operational Interface -----------eth1/2.150 IP Address --------------169.10.10.1/24 Oper. Intf -------eth1/2.14 Oper. State -------down show external-l3 bgp apic1# show external-l3 bgp flags_match : Properties in logical and concrete MOs are symmetric Tenant, vrf : t1, v1 Node Neighbor Session Status ---- ----------------------102 208.1.1.1 Flags RouteMap SourceIf Oper Peer Status ------------------------ ---------- ---------- ----------------- Allowed Self As Count: 3 TTL: 1 no (in) map200 (out) Vlan 160 vlan18 flags_match Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 420 Examples: Show Commands Examples: Show Commands show external-l3 route-map apic1# show external-l3 route-map Tenant : t1 VRF: v1 Table1: Route Map Configuration Node Routemap ----- ------------------------------------102 map200 value: Type Name Match ------- ------------------ -------------------- PfxList p1 100.100.100.0/24 Set Attributes Community 173.10.1.0/24 extended:as4-nn2:200:1 103 map1 103 map1 PfxList BD p1 b1 181.1.1.0/24 159.10.10.1/24 Metric: 100 Table 2 : Route Map Usage Node ----102 Routemap ----------map200 Protocol ---------bgp Neighbors -----------------208.1.1.1 103 map1 ospf 0.0.0.1 Operational Attributes ---------------------Pfx List: p1 100.100.100.0/24 173.10.1.0/24 ::/0 Pfx List: p1 Metric: 100 181.1.1.0/24 ::/0 Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 421 Examples: Show Commands Examples: Show Commands Cisco APIC NX-OS Style Command-Line Interface Configuration Guide 422
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project
advertisement
© 2021