Cisco APIC Troubleshooting Guide

Cisco APIC Troubleshooting Guide
Cisco APIC Troubleshooting Guide
Last Modified: 2017-01-03
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS,
INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH
THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY,
CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version
of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS.
CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT
LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS
HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network
topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional
and coincidental.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http://
www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1110R)
© 2014-2017
Cisco Systems, Inc. All rights reserved.
CONTENTS
Preface
Preface xi
Audience xi
Document Conventions xi
Related Documentation xiii
Documentation Feedback xiii
CHAPTER 1
New and Changed 1
New and Changed Information 1
CHAPTER 2
Troubleshooting Overview 3
Troubleshooting Basics 4
CHAPTER 3
Troubleshooting the Cisco APIC Cluster 5
Cluster Troubleshooting Scenarios 5
Cluster Faults 8
CHAPTER 4
Recovering Cisco APIC Passwords and Accessing Special Logins 11
Recovering the APIC Password 11
Using the Rescue-user Account to Erase the Cisco APIC Configuration Using the NX-OS Style
CLI 12
Using the Fallback Login Domain to Log in to the Local Database 12
CHAPTER 5
Cisco APIC Troubleshooting Operations 15
Shutting Down the APIC System 15
Shutting Down the APIC Controller Using the GUI 15
Using the APIC Reload Option Using the GUI 16
Controlling the LED Locator Using the GUI 17
Cisco APIC Troubleshooting Guide
iii
Contents
CHAPTER 6
Using the Cisco APIC Troubleshooting Tools 19
Enabling and Viewing ACL Contract Permit and Deny Logs 20
Enabling ACL Contract Permit Logging Using the GUI 20
Enabling ACL Contract Permit Logging Using the NX-OS CLI 21
Enabling ACL Contract Permit Logging Using the REST API 22
Enabling Taboo Contract Deny Logging Using the GUI 23
Enabling Taboo Contract Deny Logging Using the NX-OS CLI 23
Enabling Taboo Contract Deny Logging Using the REST API 24
Viewing ACL Permit and Deny Logs Using the GUI 25
Viewing ACL Permit and Deny Logs Using the REST API 25
Viewing ACL Permit and Deny Logs Using the NX-OS CLI 26
Using Atomic Counter Policies for Gathering Statistics 28
Atomic Counters 29
Atomic Counters Guidelines and Restrictions 30
Configuring Atomic Counters 30
Enabling Atomic Counters 31
Troubleshooting Using Atomic Counters with the REST API 32
Enabling and Viewing Digital Optical Monitoring Statistics 32
Enabling Digital Optical Monitoring Using the GUI 32
Enabling Digital Optical Monitoring Using the REST API 33
Viewing Digital Optical Monitoring Statistics With the GUI 35
Troubleshooting Using Digital Optical Monitoring With the REST API 35
Viewing and Understanding Health Scores 36
Health Score Types 36
Filtering by Health Score 36
Viewing Tenant Health 36
Viewing Fabric Health 37
Viewing MO Health in Visore 37
Debugging Health Scores Using Logs 37
Viewing Faults 38
Enabling Port Tracking for Uplink Failure Detection 39
Port Tracking Policy for Uplink Failure Detection 39
Port Tracking Using the GUI 40
Port Tracking Using the NX-OS CLI 40
Cisco APIC Troubleshooting Guide
iv
Contents
Port Tracking Using the REST API 41
Configuring SNMP for Monitoring and Managing Devices 41
About SNMP 41
SNMP Access Support in ACI 41
Configuring the SNMP Policy Using the GUI 42
Configuring an SNMP Trap Destination Using the GUI 43
Configuring an SNMP Trap Source Using the GUI 44
Monitoring the System Using SNMP 45
Configuring SPAN for Traffic Monitoring 45
About SPAN 45
SPAN Guidelines and Restrictions 46
Configuring a SPAN Session 46
Using Statistics 47
Viewing Statistics in the GUI 47
Switch Statistics Commands 48
Managing Statistics Thresholds Using the GUI 49
Statistics Troubleshooting Scenarios 49
Statistics Cleanup 51
Specifying Syslog Sources and Destinations 52
About Syslog 52
Creating a Syslog Destination and Destination Group 53
Creating a Syslog Source 53
Enabling Syslog to Display in NX-OS CLI Format, Using the REST API 55
Discovering Paths and Testing Connectivity with Traceroute 55
About Traceroute 56
Traceroute Guidelines and Restrictions 56
Performing a Traceroute Between Endpoints 56
Using the Troubleshooting Wizard 57
Getting Started with the Troubleshooting Wizard 57
Generating Troubleshooting Reports 59
Topology in the Troubleshooting Wizard 60
Using the Faults Troubleshooting Screen 61
Using the Drop/Statistics Troubleshooting Screen 62
Using the Contracts Troubleshooting Screen 63
Using the Events Troubleshooting Screen 64
Cisco APIC Troubleshooting Guide
v
Contents
Using the Traceroute Troubleshooting Screen 65
Using the Atomic Counter Troubleshooting Screen 66
Using the SPAN Troubleshooting Screen 66
L4 - L7 Services Validated Scenarios 67
List of APIs for Endpoint to Endpoint Connections 68
interactive API 68
createsession API 69
modifysession API 71
atomiccounter API 71
traceroute API 71
span API 72
generatereport API 73
schedulereport API 74
getreportstatus API 74
getreportslist API 75
getsessionslist API 75
getsessiondetail API 75
deletesession API 75
clearreports API 76
contracts API 77
List of APIs for Endpoint to Layer 3 External Connections 77
interactive API 78
createsession API 78
modifysession API 79
atomiccounter API 80
traceroute API 81
span API 82
generatereport API 83
schedulereport API 84
getreportstatus API 85
getreportslist API 85
getsessionslist API 85
getsessiondetail API 87
deletesession API 88
clearreports API 88
Cisco APIC Troubleshooting Guide
vi
Contents
contracts API 88
ratelimit API 89
13ext API 90
CHAPTER 7
Manually Removing Disabled Interfaces and Decommissioned Switches from the GUI 93
Manually Removing Disabled Interfaces and Decommissioned Switches from the GUI 93
CHAPTER 8
Troubleshooting Steps for Endpoint Connectivity Problems 95
Troubleshooting Endpoint Connectivity 95
Inspecting Endpoint and Tunnel Interface Status 96
Inspecting the Endpoint Status 96
Inspecting the Tunnel Interface Status 97
Connecting an SFP Module 97
CHAPTER 9
Troubleshooting EVPN Type-2 Route Advertisement 99
Troubleshooting EVPN Type-2 Route Distribution to a DCIG 99
CHAPTER 10
Performing a Rebuild of the Fabric 103
Rebuilding the Fabric 103
CHAPTER 11
Verifying IP-Based EPG Configurations 105
Verifying IP-Based EPG Configurations Using the GUI 105
Verifying IP-EPG Configurations Using Switch Commands 106
CHAPTER 12
Recovering a Disconnected Leaf 109
Recovering a Disconnected Leaf Using the REST API 109
CHAPTER 13
Determining Why a PIM Interface Was Not Created 111
A PIM Interface Was Not Created For an L3Out Interface 111
A PIM Interface Was Not Created For a Multicast Tunnel Interface 112
A PIM Interface Was Not Created For a Multicast-Enabled Bridge Domain 112
CHAPTER 14
Confirming the Port Security Installation 113
Confirming Your Port Security Installation Using Visore 113
Confirming Your Hardware Port Security Installation Using the Cisco NX-OS CLI 113
Cisco APIC Troubleshooting Guide
vii
Contents
CHAPTER 15
Troubleshooting QoS Policies 117
Troubleshooting Cisco APIC QoS Policies 117
CHAPTER 16
Determining the Supported SSL Ciphers 119
About SSL Ciphers 119
Determining the Supported SSL Ciphers Using the CLI 120
CHAPTER 17
Removing Unwanted _ui_ Objects 121
Removing Unwanted _ui_ Objects Using the REST API 123
APPENDIX A
acidiag Command 125
APPENDIX B
Configuring Export Policies for Troubleshooting 133
About Exporting Files 133
File Export Guidelines and Restrictions 133
Configuring a Remote Location 134
Configuring a Remote Location Using the GUI 134
Configuring a Remote Location Using the REST API 134
Configuring a Remote Location Using the NX-OS Style CLI 134
Sending an On-Demand Tech Support File 136
Sending an On-Demand Techsupport File Using the GUI 136
Sending an On-Demand TechSupport File Using the REST API 136
APPENDIX C
Finding the Switch Inventory 139
Finding Your Switch Inventory Using the GUI 139
Finding Your Switch Inventory Using the NX-OS CLI 140
Finding Your Switch Inventory Using the REST API 142
APPENDIX D
Cisco APIC Cluster Management 145
Expanding the Cisco APIC Cluster 145
Contracting the Cisco APIC Cluster 146
Cluster Management Guidelines 146
Expanding the APIC Cluster Size 147
Reducing the APIC Cluster Size 147
Cisco APIC Troubleshooting Guide
viii
Contents
Replacing Cisco APIC Controllers in the Cluster 148
Expanding the Cluster Examples 149
Expanding the APIC Cluster Using the GUI 149
Expanding the APIC Cluster Using the REST API 150
Contracting the Cluster Examples 150
Contracting the APIC Cluster Using the GUI 150
Contracting the APIC Cluster Using the REST API 151
Commissioning and Decommissioning Cisco APIC Controllers 152
Commissioning a Cisco APIC Controller in the Cluster Using the GUI 152
Decommissioning a Cisco APIC Controller in the Cluster Using the GUI 152
Replacing a Cisco APIC in a Cluster Using the CLI 153
APPENDIX E
Cisco APIC SSD Replacement 155
Replacing the Solid-State Drive in Cisco APIC 155
Cisco APIC Troubleshooting Guide
ix
Contents
Cisco APIC Troubleshooting Guide
x
Preface
This preface includes the following sections:
• Audience, page xi
• Document Conventions, page xi
• Related Documentation, page xiii
• Documentation Feedback, page xiii
Audience
This guide is intended for system and network engineers with a background in troubleshooting data systems,
networks, and storage systems.
Document Conventions
Command descriptions use the following conventions:
Convention
Description
bold
Bold text indicates the commands and keywords that you enter literally
as shown.
Italic
Italic text indicates arguments for which the user supplies the values.
[x]
Square brackets enclose an optional element (keyword or argument).
[x | y]
Square brackets enclosing keywords or arguments separated by a vertical
bar indicate an optional choice.
{x | y}
Braces enclosing keywords or arguments separated by a vertical bar
indicate a required choice.
Cisco APIC Troubleshooting Guide
xi
Preface
Document Conventions
Convention
Description
[x {y | z}]
Nested set of square brackets or braces indicate optional or required
choices within optional or required elements. Braces and a vertical bar
within square brackets indicate a required choice within an optional
element.
variable
Indicates a variable for which you supply values, in context where italics
cannot be used.
string
A nonquoted set of characters. Do not use quotation marks around the
string or the string will include the quotation marks.
Examples use the following conventions:
Convention
Description
screen font
Terminal sessions and information the switch displays are in screen font.
boldface screen font
Information you must enter is in boldface screen font.
italic screen font
Arguments for which you supply values are in italic screen font.
<>
Nonprinting characters, such as passwords, are in angle brackets.
[]
Default responses to system prompts are in square brackets.
!, #
An exclamation point (!) or a pound sign (#) at the beginning of a line
of code indicates a comment line.
This document uses the following conventions:
Note
Caution
Means reader take note. Notes contain helpful suggestions or references to material not covered in the
manual.
Means reader be careful. In this situation, you might do something that could result in equipment damage
or loss of data.
Cisco APIC Troubleshooting Guide
xii
Preface
Related Documentation
Warning
IMPORTANT SAFETY INSTRUCTIONS
This warning symbol means danger. You are in a situation that could cause bodily injury. Before you
work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with
standard practices for preventing accidents. Use the statement number provided at the end of each warning
to locate its translation in the translated safety warnings that accompanied this device.
SAVE THESE INSTRUCTIONS
Related Documentation
Cisco Application Centric Infrastructure (ACI) Documentation
The ACI documentation is available at the following URL: http://www.cisco.com/c/en/us/support/
cloud-systems-management/application-policy-infrastructure-controller-apic/
tsd-products-support-series-home.html.
Cisco Application Centric Infrastructure (ACI) Simulator Documentation
The Cisco ACI Simulator documentation is available at http://www.cisco.com/c/en/us/support/
cloud-systems-management/application-centric-infrastructure-simulator/tsd-products-support-series-home.html.
Cisco Nexus 9000 Series Switches Documentation
The Cisco Nexus 9000 Series Switches documentation is available at http://www.cisco.com/c/en/us/support/
switches/nexus-9000-series-switches/tsd-products-support-series-home.html.
Cisco Application Virtual Switch Documentation
The Cisco Application Virtual Switch (AVS) documentation is available at http://www.cisco.com/c/en/us/
support/switches/application-virtual-switch/tsd-products-support-series-home.html.
Cisco Application Centric Infrastructure (ACI) Integration with OpenStack Documentation
Cisco ACI integration with OpenStack documentation is available at http://www.cisco.com/c/en/us/support/
cloud-systems-management/application-policy-infrastructure-controller-apic/
tsd-products-support-series-home.html.
Documentation Feedback
To provide technical feedback on this document, or to report an error or omission, please send your comments
to [email protected] We appreciate your feedback.
Cisco APIC Troubleshooting Guide
xiii
Preface
Documentation Feedback
Cisco APIC Troubleshooting Guide
xiv
CHAPTER
1
New and Changed
• New and Changed Information, page 1
New and Changed Information
The following table provides an overview of the significant changes to this guide up to this current release.
The table does not provide an exhaustive list of all changes made to the guide or of the new features up to
this release.
Cisco APIC Release Version
Feature
Description
2.3(1e)
Audit Log for troubleshooting
faults
Viewing Faults, on page 38
-
Cisco APIC SSD Replacement
Replacing the Solid-State Drive in
Cisco APIC, on page 155
Release 2.1(1h)
Troubleshooting EVPN Type-2
Route Advertisement
Troubleshooting EVPN Type-2
Route Distribution to a DCIG
Troubleshooting QoS policies
Troubleshooting QoS Policies, on
page 117
Troubleshooting Unwanted UI
Objects
Removing Unwanted _ui_ Objects
Cisco APIC Troubleshooting Guide
1
New and Changed
New and Changed Information
Cisco APIC Release Version
Feature
Description
Release 2.0(1f)
Enabling and Viewing Digital
Optical Monitoring Statistics
Enabling and Viewing Digital
Optical Monitoring Statistics, on
page 32
Enabling and Viewing ACL
Contract Permit and Deny Logs
Enabling and Viewing ACL
Contract Permit and Deny Logs
Syslog in NX-OS-Style CLI
Format
Enabling Syslog to Display in
NX-OS CLI Format, Using the
REST API, on page 55
Port Security
Confirming the Port Security
Installation, on page 113
Layer 3 multicast with multicast Determining Why a PIM Interface
routing enabled using the Protocol Was Not Created, on page 111
Independent Multicast (PIM)
protocol
Release 1.2(2)
Cisco APIC Troubleshooting Guide
2
Port Tracking Policy for Uplink
Failure Detection
Enabling Port Tracking for Uplink
Failure Detection, on page 39
CHAPTER
2
Troubleshooting Overview
The chapters in this guide describe common troubleshooting tips for specific Cisco APIC features and provide
information about monitoring tools you can use for troubleshooting problems.
The features, issues, and tasks covered in this guide are listed below.
• _ui_ Objects—Explains how to remove unwanted _ui_ objects caused by making changes with the
Basic Mode or the NX-OS CLI before using the Advanced Mode.
• acidiag—Explains how to use the acidiag command for troubleshooting operations on the Cisco APIC.
• Cisco APIC Cluster—Explains how to diagnose cluster faults and troubleshoot common cluster issues.
For basic cluster management information, see the appendix of this guide.
• Cisco APIC Password Recovery and Emergency/Hidden Login Access—Explains how to recover
a password, how to access the rescue-user login to run troubleshooting commands, including erasing
the configuration, and how to access a hidden login domain in case of a lockout.
• Cisco APIC Troubleshooting Operations—Explains how to gather information about your switches
and how perform troubleshooting operations such as shutting down the system, shutting down the Cisco
APIC controller, reloading the APIC controller, and turning on the LED locator.
• Cisco APIC Troubleshooting Tools—Explains how to use the Cisco APIC troubleshooting tools for
monitoring traffic, debugging, and detecting issues such as traffic drops, misrouting, blocked paths,
and uplink failures.
• Endpoint Connectivity—Explains how to troubleshoot endpoint connectivity using the Cisco APIC
troubleshooting tools, such as traceroute, atomic counters, and SPAN, and how to connect an SFP
module to a new card.
Note
Information about the Cisco APIC troubleshooting tools is located in the Using the
Cisco APIC Troubleshooting Tools, on page 19 chapter.
• EVPN Type-2 Host Routes—Provides verification steps for this feature.
• Export Policies—Enables you to export statistics, tech support collections, faults and events, and to
process core files and debug data from the fabric to any external hosts in a variety of formats.
• Fabric Rebuild—Explains how to rebuild your fabric.
Cisco APIC Troubleshooting Guide
3
Troubleshooting Overview
Troubleshooting Basics
• IP-Based EPG—Explains how to verify that you have correctly configured an IP-based EPG using
the Cisco APIC GUI and using switch commands.
• Leaf Connectivity—Explains how to recover a disconnected leaf using the REST API.
• PIM Interfaces—Explains what to check when a PIM interface is not created for an L3Out, a multicast
tunnel interface, or for a multicast-enabled bridge domain.
• Port Security—Explains how to confirm your port security hardware and software installations.
• QoS—Provides specific troubleshooting scenarios for this feature.
• SSL Ciphers—Explains how to determine if an SSL cipher is supported.
• Switch Inventory—Explains how to find the switch serial and model numbers. This helps TAC
troubleshoot issues that you may experience.
This chapter contains the following:
• Troubleshooting Basics, page 4
Troubleshooting Basics
The following are basic steps for troubleshooting:
Before You Begin
• Familiarize yourself with the tools listed in Using the Cisco APIC Troubleshooting Tools, on page 19.
• Familiarize yourself with the Cisco APIC Troubleshooting Operations, on page 15.
• For issues with a specific feature, check the main contents of this guide for your feature. Troubleshooting
tips are listed per-feature.
Step 1
Step 2
Step 3
Gather information that defines the specific symptoms.
Note
In many cases, you can use the tools listed and described in the Using the Cisco APIC Troubleshooting Tools,
on page 19 chapter to gather useful troubleshooting information.
Identify all potential problems that could be causing the symptoms.
Systematically eliminate each potential problem (from most likely to least likely) until the symptoms disappear.
Note
This guide provides step-by-step instructions for confirming the installation and configuration of specific features
such as port security, endpoint connectivity, PIM, and IP-based EPGs. Following the instructions can help you
to narrow down and resolve problems you are experiencing.
Cisco APIC Troubleshooting Guide
4
CHAPTER
3
Troubleshooting the Cisco APIC Cluster
This chapter contains information about cluster faults and possible solutions for common scenarios. For
information about managing Cisco APIC clusters, see the appendix in this document.
This chapter contains the following sections:
• Cluster Troubleshooting Scenarios, page 5
• Cluster Faults, page 8
Cluster Troubleshooting Scenarios
The following table summarizes common cluster troubleshooting scenarios for the Cisco APIC.
Problem
Solution
An APIC node fails
within the cluster. For
example, node 2 of a
cluster of 5 APICs fails.
There are two available solutions:
• Leave the target size and replace the APIC.
• Reduce the cluster size to 4, decommission controller 5, and recommission
it as APIC 2.The target size remains 4, and the operational size is 4 when
the reconfigured APIC becomes active.
Note
You can add a replacement APIC to the cluster and expand the target
and operational size. For instructions on how to add a new APIC, refer
to the Cisco APIC Management, Installation, Upgrade, and Downgrade
Guide..
Cisco APIC Troubleshooting Guide
5
Troubleshooting the Cisco APIC Cluster
Cluster Troubleshooting Scenarios
Problem
Solution
A new APIC connects to Use the following commands to check for an infra (infrastructure) VLAN
the fabric and loses
mismatch:
connection to a leaf
• cat /mit/sys/lldp/inst/if-\[eth1--1\]/ctrlradj/summary—Displays the VLAN
switch.
configured on the leaf switch.
• cat /mit/sys/lldp/inst/if-\[eth1--1\]/ctrlradj/summary—Displays the infra
(infrastructure) VLANs advertised by connected APICs.
If the output of these commands shows different VLANs, the new APIC is not
configured with the correct infra (infrastructure) VLAN. To correct this issue,
follow these steps:
• Log in to the APIC using rescue-user.
Note
Admin credentials do not work because the APIC is not part of
the fabric.
• Erase the configuration and reboot the APIC using the acidiag touch setup
command.
• Reconfigure the APIC. Verify that the fabric name, TEP addresses, and
infra (infrastructure) VLAN match the APICs in the cluster.
• Reload the leaf node.
Two APICs cannot
communicate after a
reboot.
The issue can occur after the following sequence of events:
• APIC1 and APIC2 discover each other.
• APIC1 reboots and becomes active with a new ChassisID (APIC1a)
• The two APICs no longer communicate.
In this scenario, APIC1a discovers APIC2, but APIC2 is unavailable because it
is in a cluster with APIC1, which appears to be offline. As a result, APIC1a does
not accept messages from APIC2.
To resolve the issue, decommission APIC1 on APIC2, and commission APIC1
again.
A decommissioned APIC The issue can occur after the following sequence of events:
joins a cluster.
• A member of the cluster becomes unavailable or the cluster splits.
• An APIC is decommissioned.
• After the cluster recovers, the decommissioned APIC is automatically
commissioned.
To resolve the issue, decommission the APIC after the cluster recovers.
Cisco APIC Troubleshooting Guide
6
Troubleshooting the Cisco APIC Cluster
Cluster Troubleshooting Scenarios
Problem
Solution
Mismatched ChassisID
following reboot.
The issue occurs when an APIC boots with a ChassisID different from the
ChassisID registered in the cluster. As a result, messages from this APIC are
discarded.
To resolve the issue, ensure that you decommission the APIC before rebooting.
The APIC displays faults A variety of conditions can prevent a cluster from extending the
during changes to cluster OperationalClusterSize to meet the AdminstrativeClusterSize. For more
size.
information, inspect the fault and review Cluster Faults.
An APIC is unable to join The issue occurs when two APICs are configured with the same ClusterID when
a cluster.
a cluster expands. As a result, one of the two APICs cannot join the cluster and
displays an expansion-contender-chassis-id-mismatch fault.
To resolve the issue, configure the APIC outside the cluster with a new cluster
ID.
APIC unreachable in
cluster.
Check the following settings to diagnose the issue:
• Verify that fabric discovery is complete.
• Identify the switch that is missing from the fabric.
• Check whether the switch has requested and received an IP address from
an APIC.
• Verify that the switch has loaded a software image.
• Verify how long the switch has been active.
• Verify that all processes are running on the switch. For more information,
see the acidiag Command.
• Confirm that the missing switch has the correct date and time.
• Confirm that the switch can communicate with other APICs.
Cluster does not expand. The issue occurs under the following circumstances:
• The OperationalClusterSize is smaller than the number of APICs.
• No expansion contender (for example, the admin size is 5 and there is not
an APIC with a clusterID of 4.
• There is no connectivity between the cluster and a new APIC
• Heartbeat messages are rejected by the new APIC
• System is not healthy
• An unavailable appliance is carrying a data subset that is related to relocation
• Service is down on an appliance with a data subset that is related to
relocation
• Unhealthy data subset related to relocation
Cisco APIC Troubleshooting Guide
7
Troubleshooting the Cisco APIC Cluster
Cluster Faults
Problem
Solution
An APIC is down.
Check the following:
• Connectivity issue—Verify connectivity using ping.
• Interface type mismatch—Confirm that all APICs are set to in-band
communication.
• Fabric connectivity—Confirm that fabric connectivity is normal and that
fabric discovery is complete.
• Heartbeat rejected—Check the fltInfraIICIMsgSrcOutsider fault. Common
errors include operational cluster size, mismatched ChassisID, source ID
outside of the operational cluster size, source not commissioned, and fabric
domain mismatch.
Cluster Faults
The APIC supports a variety of faults to help diagnose cluster problems. The following sections describe the
two major cluster fault types.
Discard Faults
The APIC discards cluster messages that are not from a current cluster peer or cluster expansion candidate.
If the APIC discards a message, it raises a fault that contains the originating APIC's serial number, cluster ID,
and a timestamp. The following table summarizes the faults for discarded messages:
Fault
Meaning
expansion-contender-chassis-id-mismatch
The ChassisID of the transmitting APIC does not match
the ChassisID learned by the cluster for expansion.
expansion-contender-fabric-domain-mismatch
The FabricID of the transmitting APIC does not match
the FabricID learned by the cluster for expansion.
expansion-contender-id-is-not-next-to-oper-cluster-size
The transmitting APIC has an inappropriate cluster ID
for expansion. The value should be one greater than the
current OperationalClusterSize.
expansion-contender-message-is-not-heartbeat
The transmitting APIC does not transmit continuous
heartbeat messages.
fabric-domain-mismatch
The FabricID of the transmitting APIC does not match
the FabricID of the cluster.
operational-cluster-size-distance-cannot-be-bridged
The transmitting APIC has an OperationalClusterSize
that is different from that of the receiving APIC by more
than 1. The receiving APIC rejects the request.
source-chassis-id-mismatch
The ChassisID of the transmitting APIC does not match
the ChassisID registered with the cluster.
Cisco APIC Troubleshooting Guide
8
Troubleshooting the Cisco APIC Cluster
Cluster Faults
Fault
Meaning
source-cluster-id-illegal
The transmitting APIC has a clusterID value that is not
permitted.
source-has-mismatched-target-chassis-id
The target ChassisID of the transmitting APIC does not
match the Chassis ID of the receiving APIC.
source-id-is-outside-operational-cluster-size
The transmitting APIC has a cluster ID that is outside of
the OperationalClusterSize for the cluster.
source-is-not-commissioned
The transmitting APIC has a cluster ID that is currently
decommissioned in the cluster.
Cluster Change Faults
The following faults apply when there is an error during a change to the APIC cluster size.
Fault
Meaning
cluster-is-stuck-at-size-2
This fault is issued if the OperationalClusterSize remains at 2 for an
extended period. To resolve the issue, restore the cluster target size.
most-right-appliance-remains-commissioned
The last APIC within a cluster is still in service, which prevents the
cluster from shrinking.
no-expansion-contender
The cluster cannot detect an APIC with a higher cluster ID, preventing
the cluster from expanding.
service-down-on-appliance-carrying-replica-related-to-relocation The data subset to be relocated has a copy on a service that is
experiencing a failure. Indicates that there are multiple such failures
on the APIC.
unavailable-appliance-carrying-replica-related-to-relocation The data subset to be relocated has a copy on an unavailable APIC.
To resolve the fault, restore the unavailable APIC.
unhealthy-replica-related-to-relocation
The data subset to be relocated has a copy on an APIC that is not
healthy. To resolve the fault, determine the root cause of the failure.
APIC Unavailable
The following cluster faults can apply when an APIC is unavailable:
Fault
Meaning
fltInfraReplicaReplicaState
The cluster is unable to bring up a data subset.
fltInfraReplicaDatabaseState
Indicates a corruption in the data store service.
fltInfraServiceHealth
Indicates that a data subset is not fully functional.
fltInfraWiNodeHealth
Indicates that an APIC is not fully functional.
Cisco APIC Troubleshooting Guide
9
Troubleshooting the Cisco APIC Cluster
Cluster Faults
Cisco APIC Troubleshooting Guide
10
CHAPTER
4
Recovering Cisco APIC Passwords and
Accessing Special Logins
This chapter explains how to recover your Cisco APIC password, how to access the rescue-user login to run
troubleshooting commands, including the command for erasing the configuration, and how to access a hidden
login domain that allows you to log in using the local user database in case of a lockout.
This chapter contains the following sections:
• Recovering the APIC Password, page 11
• Using the Rescue-user Account to Erase the Cisco APIC Configuration Using the NX-OS Style CLI,
page 12
• Using the Fallback Login Domain to Log in to the Local Database, page 12
Recovering the APIC Password
Follow these steps to recover the APIC password.
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Create and save an empty file named "aci-admin-passwd-reset.txt".
Add the file to a USB drive. You can format the USB drive to FAT or FAT32.
Connect the USB drive to one of the rear USB ports on the Cisco APIC.
Reboot the APIC using Cisco Integrated Management Controller (CIMC) or by hard power cycling the device.
When the APIC displays the "Press any key to enter the menu" prompt, press a key to interrupt the boot process.
The APIC displays supported Linux versions. Highlight the version installed on your system and press e to edit the boot
command.
Highlight the kernel and press e to edit the command in boot sequence.
Step 8
Add the name of the empty file to the end of the command, shown as follows:
Example:
[ Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists the possible
completions of a device/filename. ESC at any time cancels. ENTER
Cisco APIC Troubleshooting Guide
11
Recovering Cisco APIC Passwords and Accessing Special Logins
Using the Rescue-user Account to Erase the Cisco APIC Configuration Using the NX-OS Style CLI
at any time accepts your changes.]
< rhgb quiet selinux=0 audit=1
aci-admin-passwd-reset
Step 9
Press Enter to save the file.
Step 10
Press b to boot the APIC.
Note
To cancel the password reset operation and return to the default boot parameters, press Esc and Enter.
Step 11
The APIC boots and prompts for a new administrator password.
Using the Rescue-user Account to Erase the Cisco APIC
Configuration Using the NX-OS Style CLI
The rescue-user is an emergency login that provides access to the APIC even when it is not in a cluster. You
can use this login to run troubleshooting commands including erasing the configuration.
Note
Step 1
Step 2
Step 3
If the APIC is part of a healthy cluster, the rescue-user account is protected with the admin password.
Access the APIC using the Cisco Integrated Management Controller (CIMC) console.
Login as rescue-user.
Note
If an admin password is in place and the APIC is logged onto the fabric, the rescue-user password is the same
as the admin password. Otherwise there is no rescue-user password.
Use the acidiag touch command to clear the configuration.
Example:
apic1# acidiag touch setup
Using the Fallback Login Domain to Log in to the Local Database
There is a hidden login domain named "fallback" that allows you to log in using the local user database in
case of lockout. The format of the username used for the authentication method is apic#fallback\\<username>.
Use the fallback login domain to log in to the local database in the GUI or log in to the fallback login domain
using the NX-OS CLI, shown as follows:
apic1(config)# aaa authentication login domain fallback
apic1(config-domain)# ?
group Set provider group for login domain
realm Specify server realm
Cisco APIC Troubleshooting Guide
12
Recovering Cisco APIC Passwords and Accessing Special Logins
Using the Fallback Login Domain to Log in to the Local Database
Or, you can use the REST API to log in to the fallback login domain, shown as follows:
• URL: https://ifav41-ifc1/api/aaaLogin.xml
• DATA:
<aaaUser name="apic#fallback\\admin"
pwd="passwordhere"/>
Cisco APIC Troubleshooting Guide
13
Recovering Cisco APIC Passwords and Accessing Special Logins
Using the Fallback Login Domain to Log in to the Local Database
Cisco APIC Troubleshooting Guide
14
CHAPTER
5
Cisco APIC Troubleshooting Operations
This chapter explains how to perform the basic troubleshooting operations and contains the following sections:
• Shutting Down the APIC System, page 15
• Shutting Down the APIC Controller Using the GUI, page 15
• Using the APIC Reload Option Using the GUI, page 16
• Controlling the LED Locator Using the GUI, page 17
Shutting Down the APIC System
This procedure explains how to shut down the APIC system.
Note
After you shut down the system, you will move it (re-locate the entire fabric) then power it up, then update
the time zone and/or NTP servers accordingly.
1 Shut down one Cisco APIC at a time by right-clicking on a controller then choosing Shutdown from the
pull-down menu.
2 Start up the APIC at the new location.
3 Check that the cluster has fully converged.
4 Proceed with the next APIC.
Before You Begin
Ensure cluster health is fully fit.
Shutting Down the APIC Controller Using the GUI
This document describes how to shut down the APIC controller.
Cisco APIC Troubleshooting Guide
15
Cisco APIC Troubleshooting Operations
Using the APIC Reload Option Using the GUI
Note
This procedure instructs on how to shut down the APIC controller only (not the entire APIC system itself).
Following this procedure causes the controller to shut down immediately. Use caution in performing a
shutdown because the only way to bring the controller back up is to do so from the actual machine. If you
need to access the machine, refer to the "Turning on the Locator LED Using the GUI" section in this
chapter.
Shut down a single APIC controller as follows:
Note
If possible, move APICs one at a time. As long as there are at least two APICs in the cluster online, there
is read/write access. If you need to relocate more than one APIC at a time, this results in one or no
remaining controllers online, and the fabric will go into a read-only mode when they are shutdown. During
this time there can be no policy changes including Endpoint moves (including Virtual Machine movement).
Once the APICs are shut down using the following procedure, relocate the controller, and power it back
up under the new rack. Then, confirm that the cluster health returns to fully fit status.
1 In the menu bar, click System.
2 In the submenu bar, click Controllers.
3 Under Controllers, click the APIC node that you would like to reload, for example, apic1 (Node-1).
4 In the right window pane, at the top of the screen, click the General tab.
5 In the right window pane, at the top of the screen and under the tabs, click the ACTIONS pull-down menu.
6 Select Shutdown from the pull-down menu to immediately reload the APIC controller.
Note
Another way to use this Shutdown option is to right-click on the APIC node (such as apic1 (Node-1), and
select Shutdown from the pull-down list.
7 Relocate the controller, then power it up.
8 Confirm cluster health returns to fully fit status.
Using the APIC Reload Option Using the GUI
This document describes how to reload the APIC controller (not the entire APIC system) using the GUI.
Reload the APIC controller as follows:
1 In the menu bar, click System.
2 In the submenu bar, click Controllers.
3 Under Controllers, click the APIC node that you would like to reload, for example, apic1 (Node-1).
4 In the right window pane, at the top of the screen, click the General tab.
5 In the right window pane, at the top of the screen and under the tabs, click the ACTIONS pull-down menu.
Cisco APIC Troubleshooting Guide
16
Cisco APIC Troubleshooting Operations
Controlling the LED Locator Using the GUI
6 Select Reload from the pull-down menu to immediately reload the APIC controller.
Note
Another way to use this Reload option is to right-click on the APIC node (such as apic1 (Node-1), and
select Reload from the pull-down list.
Controlling the LED Locator Using the GUI
This document describes how to turn on the LED locator for the APIC controller using the GUI.
Turn on (or turn off) the LED locator of the APIC controller using the GUI as follows:
1 In the menu bar, click System.
2 In the submenu bar, click Controllers.
3 Under Controllers, click the APIC node that you would like to reload, for example, apic1 (Node-1).
4 In the right window pane, at the top of the screen, click the General tab.
5 In the right window pane, at the top of the screen and under the tabs, click the ACTIONS pull-down menu.
6 Select Turn On LED Locator (or Turn Off LED Locator) from the pull-down menu.
Note
Another way to use this option is to right-click on the APIC node (such as apic1 (Node-1), and select
Turn On LED Locator (or Turn Off LED Locator) from the pull-down list.
Cisco APIC Troubleshooting Guide
17
Cisco APIC Troubleshooting Operations
Controlling the LED Locator Using the GUI
Cisco APIC Troubleshooting Guide
18
CHAPTER
6
Using the Cisco APIC Troubleshooting Tools
This chapter introduces the tools and methodology commonly used to troubleshoot problems you may
experience. These tools can assist you with monitoring traffic, debugging, and detecting issues such as traffic
drops, misrouting, blocked paths, and uplink failures. See the tools listed below for a summary overview of
the tools described in this chapter:
• ACL Contract Permit and Deny Logs—Enables the logging of packets or flows that were allowed
to be sent because of contract permit rules and the logging of packets or flows dropped because of
taboo contract deny rules.
• Atomic Counters—Enables you to gather statistics about traffic between flows for detecting drops
and misrouting in the fabric and for enabling quick debugging and isolation of application connectivity
issues.
• Digital Optical Monitoring—Enables you to view digital optical monitoring (DOM) statistics about
a physical interface.
• Health Scores—Enables you to isolate performance issues by drilling down through the network
hierarchy to isolate faults to specific managed objects (MOs).
• Port Tracking—Enables you to monitor the status of links between leaf switches and spine switches
for detecting uplink failure.
• SNMP—Simple Network Management Protocol (SNMP) enables you to remotely monitor individual
hosts (APIC or another host) and find out the state of any particular node.
• SPAN—Switchport Analizer (SPAN) enables you to perform detailed troubleshooting or to take a
sample of traffic from a particular application host for proactive monitoring and analysis.
• Statistics—Provides real-time measures of observed objects. Viewing statistics enable you to perform
trend analysis and troubleshooting.
• Syslog—Enables you to specify the minimum severity level of messages to be sent, the items to be
included in the syslog messages, and the syslog destination. The format can also be displayed in NX-OS
CLI format.
• Traceroute—Enables you to find the routes that packets actually take when traveling to their destination.
• Troubleshooting Wizard—Enables administrators to troubleshoot issues that occur during specific
time frames, which can be designated by selecting two endpoints.
This chapter contains the following sections:
Cisco APIC Troubleshooting Guide
19
Using the Cisco APIC Troubleshooting Tools
Enabling and Viewing ACL Contract Permit and Deny Logs
• Enabling and Viewing ACL Contract Permit and Deny Logs , page 20
• Using Atomic Counter Policies for Gathering Statistics, page 28
• Enabling and Viewing Digital Optical Monitoring Statistics, page 32
• Viewing and Understanding Health Scores, page 36
• Enabling Port Tracking for Uplink Failure Detection, page 39
• Configuring SNMP for Monitoring and Managing Devices, page 41
• Configuring SPAN for Traffic Monitoring, page 45
• Using Statistics, page 47
• Specifying Syslog Sources and Destinations, page 52
• Discovering Paths and Testing Connectivity with Traceroute, page 55
• Using the Troubleshooting Wizard, page 57
Enabling and Viewing ACL Contract Permit and Deny Logs
To log and/or monitor the traffic flow for a contract rule, you can enable and view the logging of packets or
flows that were allowed to be sent because of contract permit rules and the logging of packets or flows that
were dropped because of taboo contract deny rules.
Note
• ACL contract permit logging in the ACI fabric is only supported on the following switches:
◦Cisco Nexus 93180YC-EX switch
◦Cisco Nexus 93108TC-EX switch
• Using log directive on filters in management contracts is not supported. Setting the log directive will
cause zoning-rule deployment failure.
For information on contracts, see the Cisco Application Centric Infrastructure Fundamentals guide.
Enabling ACL Contract Permit Logging Using the GUI
The following steps show how to enable Contract permit logging using the GUI:
Cisco APIC Troubleshooting Guide
20
Using the Cisco APIC Troubleshooting Tools
Enabling ACL Contract Permit Logging Using the NX-OS CLI
Note
The tenant that contains the permit logging is the tenant that contains the VRF that the EPG is associated
to. This will not necessarily be the same tenant as the EPG or its associated contracts.
Step 1
Step 2
On the menu bar, choose Tenants > <tenant name>.
In the Navigation pane, expand Security Policies.
Step 3
Step 4
Right-click Contracts and choose Create Contract.
In the Create Contract dialog box, perform the following actions:
a) In the Name field, type the name for the contract.
b) In the Scope field, choose the scope for it (VRF, Tenant, or Global).
c) Optional. Set the target DSCP or QoS class to be applied to the contract.
d) Expand Subjects.
Step 5
Step 6
Step 7
Step 8
In the Create Contract Subject dialog box, perform the following actions:
Type the name of the subject and an optional description.
Optional. From the drop-down list for the target DSCP, select the DSCP to be applied to the subject.
Leave Apply Both Directions checked, unless you want the contract to only be applied from the consumer to the provider,
instead of in both directions.
Leave Reverse Filter Ports checked if you unchecked Apply Both Directions to swap the Layer 4 source and destination
ports so that the rule is applied from the provider to the consumer.
Expand Filter Chain.
In the Name drop-down list, choose an option; for example, click arp, default, est, or icmp.
In the Directives drop-down list, click log.
Click Update.
Click OK.
Click SUBMIT.
Logging is enabled for this contract.
Step 9
Step 10
Step 11
Step 12
Step 13
Step 14
Step 15
Enabling ACL Contract Permit Logging Using the NX-OS CLI
The following example shows how to enable Contract permit logging using the NX-OS CLI.
Step 1
To enable logging of packets or flows that were allowed to be sent because of Contract permit rules, use the following
commands:
configure
tenant <tenantName>
contract <contractName> type <permit>
subject <subject Name>
access-group <access-list> <in/out/both> log
Cisco APIC Troubleshooting Guide
21
Using the Cisco APIC Troubleshooting Tools
Enabling ACL Contract Permit Logging Using the REST API
Example:
For example:
apic1# configure
apic1(config)# tenant BDMode1
apic1(config-tenant)# contract Logicmp type permit
apic1(config-tenant-contract)# subject icmp
apic1(config-tenant-contract-subj)# access-group arp both log
Step 2
To disable the permit logging use the no form of the access-group command; for example, use the no
arp both log command.
access-group
Enabling ACL Contract Permit Logging Using the REST API
The following example shows you how to enable Contract permit logging using the REST API.
To enable Contract permit logging, post data such as the following example:
POST https://192.0.20.123/api/node/mo/uni/tn-sgladwin_t1/brc-ICMP_Contract.json
{
"vzBrCP":{
"attributes":{
"dn" : "uni/tn-sgladwin_t1/brc-ICMP_Contract",
"name" : "ICMP_Contract",
"rn" : "brc-ICMP_Contract","
"status" : "created"},
"children":[{
"vzSubj":{
"attributes":{
"dn" : "uni/tn-sgladwin_t1/brc-ICMP_Contract/subj-Permit_Contract_ICMP",
"name" : "Permit_Contract_ICMP","rn":"subj-Permit_Contract_ICMP",
"status":"created"},
"children":[{
"vzRsSubjFiltAtt":{
"attributes":{
"status":"created,modified",
"tnVzFilterName":"icmp",
"directives":"log"},
"children":[]}}]}}]}}
response: {"totalCount":"0","imdata":[]}
Cisco APIC Troubleshooting Guide
22
Using the Cisco APIC Troubleshooting Tools
Enabling Taboo Contract Deny Logging Using the GUI
Enabling Taboo Contract Deny Logging Using the GUI
The following steps show how to enable Taboo Contract deny logging using the GUI:
Step 1
Step 2
On the menu bar, choose Tenants > <tenant name>.
In the Navigation pane, expand Security Policies.
Step 3
Step 4
Right-click Taboo Contracts and choose Create Taboo Contract.
In the Create Taboo Contract dialog box, perform the following actions to specify the Taboo contract:
a) In the Name field, type the name for the contract.
b) Optional. In the Description field, type a description of the Taboo contract.
c) Expand Subjects.
Step 5
In the Create Taboo Contract Subject dialog box, perform the following actions:
a) In the Specify Identity of Subject area, type a name and optional description.
b) Expand Filters.
c) From the Name drop-down list, choose one of the following values:<tenant_name>/arp, <tenant_name>/default,
<tenant_name>/est, <tenant_name>/icmp, or Create Filter.
Note
If you chose Create Filter, in the Specify Filter Identity Area, perform the following actions to specify criteria
for the ACL Deny rule:
1 Type a name and optional description.
2 Expand Entries, type a name for the rule, and choose the ACL Deny criteria for the rule.
3 Click Update.
4 Click Submit.
Step 6
Step 7
Step 8
Step 9
In the Directives drop-down list, click log.
Click Update.
Click OK.
Click SUBMIT.
Logging is enabled for this Taboo contract.
Enabling Taboo Contract Deny Logging Using the NX-OS CLI
The following example shows how to enable Taboo Contract deny logging using the NX-OS CLI.
Step 1
To enable logging of packets or flows dropped because of Taboo Contract deny rules, use the following commands:
configure
tenant <tenantName>
contract <contractName> type <deny>
Cisco APIC Troubleshooting Guide
23
Using the Cisco APIC Troubleshooting Tools
Enabling Taboo Contract Deny Logging Using the REST API
subject <subject Name>
access-group <access-list> <both> log
Example:
For example:
apic1# configure
apic1(config)# tenant BDMode1
apic1(config-tenant)# contract dropFTP type deny
apic1(config-tenant-contract)# subject dropftp
apic1(config-tenant-contract-subj)# access-group https both log
Step 2
To disable the deny logging use the no form of the access-group command; for example, use the no
https both log command.
access-group
Enabling Taboo Contract Deny Logging Using the REST API
The following example shows you how to enable Taboo Contract deny logging using the REST API.
To enable ACL deny logging, post data such as the follow example:
POST https://192.0.20.123/api/node/mo/uni/tn-sgladwin_t1/taboo-TCP_Taboo_Contract.json
{
"vzTaboo":{
"attributes":{
"dn":"uni/tn-sgladwin_t1/taboo-TCP_Taboo_Contract",
"name":"TCP_Taboo_Contract",
"rn":"taboo-TCP_Taboo_Contract",
"status":"created"},
"children":[{
"vzTSubj":{
"attributes":{
"dn":"uni/tn-sgladwin_t1/taboo-TCP_Taboo_Contract/tsubj-TCP_Filter_Subject",
"name":"TCP_Filter_Subject",
"rn":"tsubj-TCP_Filter_Subject",
"status":"created"},
"children":[{
"vzRsDenyRule":{
"attributes":{
"tnVzFilterName":"TCP_Filter",
"directives":"log",
"status":"created"},
"children":[]}}]}}]}}
response: {"totalCount":"0","imdata":[]}
Cisco APIC Troubleshooting Guide
24
Using the Cisco APIC Troubleshooting Tools
Viewing ACL Permit and Deny Logs Using the GUI
Viewing ACL Permit and Deny Logs Using the GUI
The following steps show how to view ACL permit and deny logs (if they are enabled) for traffic flows, using
the GUI:
Step 1
Step 2
On the menu bar, choose Tenants > <tenant name>.
In the Navigation pane, click on Tenant <tenant name>.
Step 3
In the Tenants <tenant name> Work pane, click the Operational tab.
Step 4
Under the Operational tab, click the Flows tab.
Under the Flows tab, click one of the tabs to view log data for Layer 2 permit logs (L2 Permit) Layer 3 permit logs (L3
Permit, Layer 2 deny logs (L2 Drop), or Layer 3 deny logs (L3 Drop). On each tab, you can view ACL logging data,
if traffic is flowing. The data points differ according to the log type and ACL rule; for example, the following data points
are included for L3 Permit logs:
• Timestamp
• VRF
• Source IP address
• Destination IP address
• Protocol
• Source Port
• Destination Port
• Source MAC address
• Destination MAC address
• Node (switch where data came from)
• Source interface
• VLAN
• VRF encapsulation
Note
You can also use the Packets tab (next to the Flows tab) to access ACL logs for groups of packets (up to 10)
with the same signature, source and destination. You can see what type of packets are being sent and which are
being dropped.
Viewing ACL Permit and Deny Logs Using the REST API
The following example shows how to view permit and deny log data for traffic flows, using the REST API:
Cisco APIC Troubleshooting Guide
25
Using the Cisco APIC Troubleshooting Tools
Viewing ACL Permit and Deny Logs Using the NX-OS CLI
Before You Begin
You must enable permit or deny logging, before you can view ACL contract permit and deny log data.
Send the following query using the REST API:
GET
https://apic-ip-address/api/node/mo/uni/tn-sgladwin_t1.json?rsp-subtree-include=stats&rsp-subtree-class=fvOverallHealthHist15min
{
"totalCount":"1",
"imdata":[{
"fvTenant":{
"attributes":{
"childAction":"",
"descr":"",
"dn":"uni/tn-sgladwin_t1",
"lcOwn":"local",
"modTs":"2016-06-22T15:46:30.745+00:00",
"monPolDn":"uni/tn-common/monepg-default",
"name":"sgladwin_t1",
"ownerKey":"",
"ownerTag":"",
"status":"",
"uid":"15374"
}}}]}
Viewing ACL Permit and Deny Logs Using the NX-OS CLI
The following steps show how to view ACL log details using the NX-OS CLI show acllog command.
The full syntax for the command is show acllog {permit | drop} l3 {pkt | flow} tenant <tenant name> vrf
<vrf name> srcip <source ip> dstip <destination ip> srcport <source port> dstport <destination port>
protocol <protocol> srcintf <source interface> start-time <startTime> end-time <endTime>
Step 1
The following example shows how to use the show acllog permit l3 pkt tenant <tenant name> vrf <vrf name> [detail]
command to display detailed information about the common VRF ACL Layer 3 permit packets that were sent:
apic1# show acllog permit l3 pkt tenant common vrf default detail
acllog permit l3 packets detail:
srcIp
: 10.2.0.19
dstIp
: 10.2.0.16
protocol
: udp
srcPort
: 13124
dstPort
: 4386
srcIntf
: port-channel5
vrfEncap
: VXLAN: 2097153
pktLen
: 112
srcMacAddr : 00:00:15:00:00:28
Cisco APIC Troubleshooting Guide
26
Using the Cisco APIC Troubleshooting Tools
Viewing ACL Permit and Deny Logs Using the NX-OS CLI
dstMacAddr : 00:00:12:00:00:25
timeStamp : 2015-03-17T21:31:14.383+00:00
Step 2
The following example shows how to use the show acllog command to display information about common VRF Layer
3 UDP packets with source IP address 10.2.0.19, destination IP address 10.2.0.16, source port 13124, destination port
4386, source interface port-channel15, start -time 2015-03-17T21:00:00, and end-time 2015-03-18T00:00:00:
apic1# show acllog permit l3 pkt tenant common vrf copy srcip 10.2.0.19 dstip 10.2.0.16 srcport 13124
dstport 4386
protocol 17 srcintf port-channel5 start-time 2015-03-17T21:00:00 end-time 2015-03-18T00:00:00
acllog Permit L3 Packets
srcIp
dstIp
protocol srcport
dstport
Node
srcIntf
vrfEncap
pktLen
timeStamp
--------------------------------- --------------------------------------- -------------10.2.0.19
10.2.0.16
udp
13124
4386
101
port-channel5
VXLAN: 2097153
112
2015-03-17T21:
31:14.383+00:00
Step 3
The following example shows how to use the show acllog permit l2 pkt tenant <tenant name> vrf <vrf name>
[detail] command to view detailed information about default VRF Layer 2 packets that were sent:
apic1# show acllog permit l2 pkt tenant common vrf default detail
acllog permit l2 packets detail:
srcIntf
: port-channel5
pktLen
: 1
srcMacAddr : 00:00:66:00:00:66
dstMacAddr : 00:00:89:00:00:00
timeStamp : 2015-03-17T21:31:14.383+00:00
Step 4
The following example shows how to use the show acllog permit l2 pkt tenant <tenant name> vrf <vrf name> srcintf
<s interface> command to view information about default VRF Layer 2 packets sent from interface port-channel15:
apic1# show acllog permit l2 pkt tenant common vrf default srcintf port-channel5
acllog permit L2 Packets
Node
srcIntf
pktLen
timeStamp
-------------- -------------- -------- -------------port-channel5
1
2015-03-17T21:
31:14.383+00:00
Step 5
The following example shows how to use the show acllog drop l3 pkt tenant <tenant name> vrf <vrf name> [detail]
command to show detailed information about packets that were dropped due to an ACL deny rule:
apic1# show acllog drop l3 pkt tenant common vrf copy detail
acllog drop l3 packets detail:
srcIp
: 10.2.0.19
dstIp
: 10.2.0.16
protocol
: udp
srcPort
: 13124
dstPort
: 4386
Cisco APIC Troubleshooting Guide
27
Using the Cisco APIC Troubleshooting Tools
Using Atomic Counter Policies for Gathering Statistics
srcIntf
vrfEncap
pktLen
srcMacAddr
dstMacAddr
timeStamp
Step 6
:
:
:
:
:
:
port-channel5
VXLAN: 2097153
112
00:00:15:00:00:28
00:00:12:00:00:25
2015-03-17T21:31:14.383+00:00
The following example shows how to use the show acllog drop l3 flow tenant <tenant name> vrf <vrf name> srcip
<source ip> dstip <dst ip> srcport <src port> dstport <dst port> protocol <protocol> srcintf <src intf> command
to show information about UDP packets that were dropped, that had source IP address 10.2.0.19, destination IP address
10.2.0.16, source port 13124, destination port 4386, and souce interface port-channel15:
apic1# show acllog drop l3 pkt tenant common vrf copy srcip 10.2.0.19 dstip 10.2.0.16 srcPort 13124
dstPort 4386 protocol 17 srcintf port-channel5
acllog drop L3 Packets
srcIp
dstIp
protocol srcPort
dstPort
Node
srcIntf
vrfEncap
pktLen
timeStamp
-------------- -------------- -------- -------- -------- -------------- --------------------------- -------- -------------10.2.0.19
10.2.0.16
udp
13124
4386
port-channel5
VXLAN:
2097153
112
2015-03-17T21: 31:14.383+00:00
Step 7
The following example shows how to use the show acllog drop l2 pkt tenant <tenant name> vrf <vrf name> [detail]
command to view detailed information about packets that were dropped because of an ACL deny rule:
apic1# show acllog drop l2 pkt tenant common vrf copy detail
acllog drop l2 packets detail:
srcIntf
: port-channel87
pktLen
: 1122
srcMacAddr : 00:00:11:00:00:11
dstMacAddr : 11:00:32:00:00:33
timeStamp : 2015-03-17T21:31:14.383+00:00
Step 8
The following example shows how to use the show acllog drop l2 pkt tenant <tenant name> vrf <vrf name> srcintf
<srcintf name> command to view information about packets dropped that originated from a specific interface:
apic1# show acllog drop l2 pkt tenant common vrf copy srcintf port-channel87
acllog drop L2 Packets
Node
srcIntf
pktLen
timeStamp
-------------- -------------- -------- -------------port-channel87
1122
2015-03-17T21:
31:14.383+00:00
Using Atomic Counter Policies for Gathering Statistics
Atomic counter policies enable you to gather statistics about your traffic between a combination of endpoints,
endpoint groups, external interfaces, and IP addresses. The information gathered enables you to detect drops
and misrouting in the fabric, which enables you to perform quick debugging and to isolate application
connectivity issues.
Cisco APIC Troubleshooting Guide
28
Using the Cisco APIC Troubleshooting Tools
Atomic Counters
Atomic Counters
Atomic Counters are useful for troubleshooting connectivity between endpoints, EPGs, or an application
within the fabric. A user reporting application may be experiencing slowness, or atomic counters may be
needed for monitoring any traffic loss between two endpoints. One capability provided by atomic counters is
the ability to place a trouble ticket into a proactive monitoring mode, for example when the problem is
intermittent, and not necessarily happening at the time the operator is actively working the ticket.
Atomic counters can help detect packet loss in the fabric and allow the quick isolation of the source of
connectivity issues. Atomic counters require NTP to be enabled on the fabric.
Leaf-to-leaf (TEP to TEP) atomic counters can provide the following:
• Counts of drops, admits, and excess packets
• Short-term data collection such as the last 30 seconds, and long-term data collection such as 5 minutes,
15 minutes, or more
• A breakdown of per-spine traffic (available when the number of TEPs, leaf or VPC, is less than 64)
• Ongoing monitoring
Leaf-to-leaf (TEP to TEP) atomic counters are cumulative and cannot be cleared. However, because 30 second
atomic counters reset at 30 second intervals, they can be used to isolate intermittent or recurring problems.
Tenant atomic counters can provide the following:
• Application-specific counters for traffic across the fabric, including drops, admits, and excess packets
• Modes include the following:
• Endpoint to endpoint MAC address, or endpoint to endpoint IP address. Note that a single target endpoint
could have multiple IP addresses associated with it.
• EPG to EPG with optional drill down
• EPG to endpoint
• EPG to * (any)
• Endpoint to external IP address
Note
Atomic counters track the amount packets of between the two endpoints and use this as a measurement.
They do not take into account drops or error counters in a hardware level.
Dropped packets are calculated when there are less packets received by the destination than transmitted by
the source.
Excess packets are calculated when there are more packets received by the destination than transmitted by
the source.
Cisco APIC Troubleshooting Guide
29
Using the Cisco APIC Troubleshooting Tools
Atomic Counters Guidelines and Restrictions
Atomic Counters Guidelines and Restrictions
• Use of atomic counters is not supported when the endpoints are in different tenants or in different contexts
(VRFs) within the same tenant.
• In pure layer 2 configurations where the IP address is not learned (the IP address is 0.0.0.0),
endpoint-to-EPG and EPG-to-endpoint atomic counter policies are not supported. In these cases,
endpoint-to-endpoint and EPG-to-EPG policies are supported. External policies are virtual routing and
forwarding (VRF)-based, requiring learned IP addresses, and are supported.
• When the atomic counter source or destination is an endpoint, the endpoint must be dynamic and not
static. Unlike a dynamic endpoint (fv:CEp), a static endpoint (fv:StCEp) does not have a child object
(fv:RsCEpToPathEp) that is required by the atomic counter.
• In a transit topology, where leaf switches are not in full mesh with all spine switches, then leaf-to-leaf
(TEP to TEP) counters do not work as expected.
• For leaf-to-leaf (TEP to TEP) atomic counters, once the number of tunnels increases the hardware limit,
the system changes the mode from trail mode to path mode and the user is no longer presented with
per-spine traffic.
• The atomic counter does not count spine proxy traffic.
• Packets dropped before entering the fabric or before being forwarded to a leaf port are ignored by atomic
counters.
• Packets that are switched in the hypervisor (same Port Group and Host) are not counted.
• Atomic counters require an active fabric Network Time Protocol (NTP) policy.
• Atomic counters work for IPv6 sources and destinations but configuring source and destination IP
addresses across IPv4 and IPv6 addresses is not allowed.
• An atomic counter policy configured with fvCEp as the source and/or destination counts only the traffic
that is from/to the MAC and IP addresses that are present in the fvCEp managed objects (MOs). If the
fvCEp MO has an empty IP address field, then all traffic to/from that MAC address would be counted
regardless of the IP address. If the APIC has learned multiple IP addresses for an fvCEp, then traffic
from only the one IP address in the fvCEp MO itself is counted as previously stated. In order to configure
an atomic counter policy to/from a specific IP address, use the fvIp MO as the source and/or destination.
• If there is an fvIp behind an fvCEp, you must add fvIP-based policies and not fvCEp-based policies.
Configuring Atomic Counters
Step 1
Step 2
Step 3
In the menu bar, click Tenants.
In the submenu bar, click the desired tenant.
In the Navigation pane, expand the tenant and expand Troubleshoot Policies.
Step 4
Under Troubleshoot Policies, expand Atomic Counter Policy and choose a traffic topology.
Cisco APIC Troubleshooting Guide
30
Using the Cisco APIC Troubleshooting Tools
Enabling Atomic Counters
You can measure traffic between a combination of endpoints, endpoint groups, external interfaces, and IP addresses.
Step 5
Step 6
Right-click the desired topology and choose Add topology Policy to open an Add Policy dialog box.
In the Add Policy dialog box, perform the following actions:
a) In the Name field, enter a name for the policy.
b) choose or enter the identifying information for the traffic source.
The required identifying information differs depending on the type of source (endpoint, endpoint group, external
interface, or IP address).
c) choose or enter the identifying information for the traffic destination.
d) (Optional) (Optional) In the Filters table, click the + icon to specify filtering of the traffic to be counted.
In the resulting Create Atomic Counter Filter dialog box, you can specify filtering by the IP protocol number
(TCP=6, for example) and by source and destination IP port numbers.
e) Click Submit to save the atomic counter policy.
Step 7
In the Navigation pane, under the selected topology, choose the new atomic counter policy.
The policy configuration is displayed in the Work pane.
Step 8
In the Work pane, click the Operational tab and click the Traffic subtab to view the atomic counter statistics.
Enabling Atomic Counters
To enable using atomic counters to detect drops and misrouting in the fabric and enable quick debugging and
isolation of application connectivity issues, create one or more tenant atomic counter policies, which can be
one of the following types:
• EP_to_EP—Endpoint to endpoint (dbgacEpToEp)
• EP_to_EPG—Endpoint to endpoint group (dbgacEpToEpg)
• EP_to_Ext—Endpoint to external IP address (dbgacEpToExt)
• EPG_to_EP—Endpoint group to endpoint(dbgacEpgToEp)
• EPG_to_EPG—Endpoint group to endpoing group (dbgacEpgToEpg)
• EPG_to_IP—Endpoint group to IP address (dbgacEpgToIp)
• Ext_to_EP—External IP address to endpoint (dbgacExtToEp)
• IP_to_EPG—IP address to endpoint group (dbgacIpToEpg)
• Any_to_EP—Any to endpoint (dbgacAnyToEp)
• EP_to_Any—Endpoint to any (dbgacEpToAny)
Step 1
To create an EP_to_EP policy using the REST API, use XML such as the following example:
Example:
<dbgacEpToEp name="EP_to_EP_Policy" ownerTag="" ownerKey=""
dn="uni/tn-Tenant64/acEpToEp-EP_to_EP_Policy" descr="" adminSt="enabled">
Cisco APIC Troubleshooting Guide
31
Using the Cisco APIC Troubleshooting Tools
Troubleshooting Using Atomic Counters with the REST API
<dbgacFilter name="EP_to_EP_Filter" ownerTag="" ownerKey="" descr=""
srcPort="https" prot="tcp" dstPort="https"/>
</dbgacEpToEp>
Step 2
To create an EP_to_EPG policy using the REST API, use XML such as the following example:
Example:
<dbgacEpToEpg name="EP_to_EPG_Pol" ownerTag="" ownerKey=""
dn="uni/tn-Tenant64/epToEpg-EP_to_EPG_Pol" descr="" adminSt="enabled">
<dbgacFilter name="EP_to_EPG_Filter" ownerTag="" ownerKey="" descr=""
srcPort="http" prot="tcp" dstPort="http"/>
<dbgacRsToAbsEpg tDn="uni/tn-Tenant64/ap-VRF64_app_prof/epg-EPG64"/>
</dbgacEpToEpg>
Troubleshooting Using Atomic Counters with the REST API
Step 1
To get a list of the endpoint-to-endpoint atomic counters deployed within the fabric and the associated details such as
dropped packet statistics and packet counts, use the dbgEpToEpTsIt class in XML such as the following example:
Example:
https://apic-ip-address/api/node/class/dbgEpToEpRslt.xml
Step 2
To get a list of external IP-to-endpoint atomic counters and the associated details, use the dbgacExtToEp class in XML
such as the following example:
Example:
https://apic-ip-address/api/node/class/dbgExtToEpRslt.xml
Enabling and Viewing Digital Optical Monitoring Statistics
Real-time digital optical monitoring (DOM) data is collected from SFPs, SFP+, and XFPs periodically and
compared with warning and alarm threshold table values. The DOM data collected are transceiver transmit
bias current, transceiver transmit power, transceiver receive power, and transceiver power supply voltage.
Enabling Digital Optical Monitoring Using the GUI
Before you can view digital optical monitoring (DOM) statistics about a physical interface, enable DOM on
the leaf or spine interface, using a switch policy, associated to a policy group.
Cisco APIC Troubleshooting Guide
32
Using the Cisco APIC Troubleshooting Tools
Enabling Digital Optical Monitoring Using the REST API
To enable DOM using the GUI:
Step 1
Step 2
On the menu bar, choose Fabric > Fabric Policies.
In the Navigation pane, expand Switch Policies > Policies > Fabric Node Controls.
Step 3
Step 4
Expand Fabric Node Controls to see a list of existing policies.
In the Work pane, click the ACTIONS drop-down menu and select Create Fabric Node Control.
The Create Fabric Node Control dialog box appears.
In the Create Fabric Node Control dialog box, perform the following actions:
a) In the Name field, enter a name for the policy.
b) Optional. In the Description field, enter a description of the policy.
c) Check the box next to Enable DOM.
Step 5
Step 6
Step 7
Step 8
Step 9
Step 10
Click SUBMIT to create the policy.
Now you can associate this policy to a policy group and a profile, as described in the following steps.
In the Navigation pane, expand Switch Policies > Policy Groups.
In the Work pane, click the ACTIONS drop-down menu and select Create Leaf Switch Policy Group (for a spine,
Create Spine Switch Policy Group.
The Create Leaf Switch Policy Group or Create Spine Switch Policy Group dialog box appears.
In the dialog box, perform the following actions:
a) In the Name field, enter a name for the policy group.
b) From the Node Control Policy drop-down menu, choose either an existing policy (such as the one you just created)
or a new one by selecting Create Fabric Node Control.
c) Click SUBMIT.
Attach the policy group you created to a switch as follows:
a) In the Navigation pane, expand Switch Policies > Profiles.
b) In the Work pane, click the ACTIONS drop-down menu and select Create Leaf Switch Profile or Create Spine
Switch Profile, as appropriate.
c) In the dialog box, enter a name for the profile in the Name field.
d) Add the name of the switch you want associated with the profile under Switch Associations.
e) From the Blocks pull-down menu, check the boxes next to the applicable switches.
f) From the Policy Group pull-down menu, select the policy group you created earlier.
g) Click UPDATE, then click SUBMIT.
Enabling Digital Optical Monitoring Using the REST API
Before you can view digital optical monitoring (DOM) statistics about a physical interface, enable DOM on
the interface.
To enable DOM using the REST API:
Step 1
Create a fabric node control policy (fabricNodeControlPolicy) as in the following example:
Cisco APIC Troubleshooting Guide
33
Using the Cisco APIC Troubleshooting Tools
Enabling Digital Optical Monitoring Using the REST API
<fabricNodeControl dn="uni/fabric/nodecontrol-testdom" name="testdom" control="1"
rn="nodecontrol-testdom" status="created" />
Step 2
Associate a fabric node control policy to a policy group as follows:
<?xml version="1.0" encoding="UTF-8" ?>
<fabricLeNodePGrp dn="uni/fabric/funcprof/lenodepgrp-nodegrp2" name="nodegrp2"
rn="lenodepgrp-nodegrp2" status="created,modified" >
<fabricRsMonInstFabricPol tnMonFabricPolName="default" status="created,modified" />
<fabricRsNodeCtrl tnFabricNodeControlName="testdom" status="created,modified" />
</fabricLeNodePGrp>
Step 3
Associate a policy group to a switch (in the following example, the switch is 103) as follows:
<?xml version="1.0" encoding="UTF-8" ?>
<fabricLeafP>
<attributes>
<dn>uni/fabric/leprof-leafSwitchProfile</dn>
<name>leafSwitchProfile</name>
<rn>leprof-leafSwitchProfile</rn>
<status>created,modified</status>
</attributes>
<children>
<fabricLeafS>
<attributes>
<dn>uni/fabric/leprof-leafSwitchProfile/leaves-test-typ-range</dn>
<type>range</type>
<name>test</name>
<rn>leaves-test-typ-range</rn>
<status>created,modified</status>
</attributes>
<children>
<fabricNodeBlk>
<attributes>
<dn>uni/fabric/leprof-leafSwitchProfile/leaves-test-typ-range/nodeblk-09533c1d228097da</dn>
<from_>103</from_>
<to_>103</to_>
<name>09533c1d228097da</name>
<rn>nodeblk-09533c1d228097da</rn>
<status>created,modified</status>
</attributes>
</fabricNodeBlk>
</children>
<children>
<fabricRsLeNodePGrp>
<attributes>
<tDn>uni/fabric/funcprof/lenodepgrp-nodegrp2</tDn>
<status>created</status>
</attributes>
</fabricRsLeNodePGrp>
</children>
</fabricLeafS>
Cisco APIC Troubleshooting Guide
34
Using the Cisco APIC Troubleshooting Tools
Viewing Digital Optical Monitoring Statistics With the GUI
</children>
</fabricLeafP>
Viewing Digital Optical Monitoring Statistics With the GUI
To view DOM statistics using the GUI:
Before You Begin
You must have previously enabled digital optical monitoring (DOM) statistics for an interface, before you
can view the DOM statistics for it.
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
In the Menu bar, choose Fabric and Inventory.
In the Navigation pane, expand the Pod and Leaf node where the physical interface you are investigating is located.
Expand Interfaces.
Expand Physical Interfaces.
Expand the physical interface you are investigating.
Choose DOM Stats.
DOM statitistics are displayed for the interface.
Troubleshooting Using Digital Optical Monitoring With the REST API
To view DOM statistics using an XML REST API query:
Before You Begin
You must have previously enabled digital optical monitoring (DOM) on an interface, before you can view
the DOM statistics for it.
The following example shows how to view DOM statistics on a physical interface, eth1/25 on node-104, using a REST
API query:
GET https://apic-ip-address/api/node/mo/topology/pod-1/node-104/sys/phys-[eth1/25]/phys/domstats.xml?
query-target=children&target-subtree-class=ethpmDOMRxPwrStats&subscription=yes
The following response is returned:
response : {
"totalCount":"1",
"subscriptionId":"72057611234705430",
"imdata":[
{"ethpmDOMRxPwrStats":{
"attributes":{
"alert":"none",
"childAction":"",
Cisco APIC Troubleshooting Guide
35
Using the Cisco APIC Troubleshooting Tools
Viewing and Understanding Health Scores
"dn":"topology/pod-1/node-104/sys/phys[eth1/25]/phys/domstats/rxpower",
"hiAlarm":"0.158490",
"hiWarn":"0.079430",
"loAlarm":"0.001050",
"loWarn":"0.002630",
"modTs":"never",
"status":"",
"value":"0.139170"}}}]}
Viewing and Understanding Health Scores
The APIC uses a policy model to combine data into a health score. Health scores can be aggregated for a
variety of areas such as for infrastructure, applications, or services. The health scores enable you to isolate
performance issues by drilling down through the network hierarchy to isolate faults to specific managed
objects (MOs). You can view network health by viewing the health of an application (by tenant) or by the
health of a leaf switch (by pod).
For more information about health scores, faults, and health score calculation see the Cisco APIC Fundamentals
Guide.
Health Score Types
The APIC supports the following health score types:
• System—Summarizes the health of the entire network.
• Leaf—Summarizes the health of leaf switches in the network. Leaf health includes hardware health of
the switch including fan tray, power supply, and CPU.
• Tenant—Summarizes the health of a tenant and the tenant’s applications.
Filtering by Health Score
You can filter health scores using the following tools:
• Health Scroll Bar—You can use the health scroll bar to dictate which objects are visible; lowering the
score allows you to see only objects with a degraded health score.
• Displaying Degraded Health Scores—To display only the degraded health scores, click the Gear icon
and choose Show only degraded health score.
Viewing Tenant Health
To view application health, click Tenants > tenant-name in the menu bar, then click the tenant name in the
Navigation pane. The GUI displays a summary of the tenant's health including applications and EPGs. To
drill down on the tenant configuration, double-click the health score.
Cisco APIC Troubleshooting Guide
36
Using the Cisco APIC Troubleshooting Tools
Viewing Fabric Health
For a health summary, click the Health tab in the Work pane. This view of the network displays health scores
and relationships between MOs in the network so that you can isolate and resolve performance issues. For
example, a common sequence of managed objects in the tenant context is Tenant> Application profile >
Application EPG > EPP > Fabric location > EPG to Path Attachment > Network Path Endpoint >
Aggregation Interface > Aggregated Interface > Aggregated Member Interface.
Viewing Fabric Health
To view fabric health, click Fabric in the menu bar. In the navigation pane, choose a pod. The GUI displays
a summary of the pod health including nodes. To drill down on part of the fabric configuration, double-click
the health score.
For a health summary, click the Health tab in the work pane. This view of the network displays health scores
and relationships between MOs in the network so that you can isolate and resolve performance issues. For
example, a common sequence of managed objects in the fabric context is Pod > Leaf> Chassis> Fan tray
slot> Line module slot > Line module > Fabric Port > Layer 1 Physical Interface Configuration >
Physical Interface Runtime State.
Note
Fabric issues, such as physical network problems, can impact tenant performance when MOs are directly
related.
Viewing MO Health in Visore
To view the health of an MO in Visore, click the H icon.
Use the following MOs to display health information:
• health:Inst
• health:NodeInst
• observer:Node
• observer:Pod
For more information about Visore, see the Cisco Application Centric Infrastructure Fundamentals guide.
Debugging Health Scores Using Logs
You can use the following log files to debug health scores on the APIC:
• svc_ifc_eventmgr.log
• svc_ifc_observer.log
Check the following items when debugging health scores using logs:
• Verify the source of the syslog (fault or event).
• Check whether a syslog policy is configured on the APIC.
Cisco APIC Troubleshooting Guide
37
Using the Cisco APIC Troubleshooting Tools
Viewing Faults
• Check whether the syslog policy type and severity is set correctly.
• You can specify a syslog destination of console, file, RemoteDest, or Prof. ForRemoteDest, ensure that
the syslog server is running and reachable.
Viewing Faults
The steps below explain where to view fault information.
Step 1
Go to a faults window:
• System Faults—From the menu bar, click System > Faults.
• Tenant Faults—From the menu bar:
1 Click Tenants > tenant-name.
2 From the Navigation pane, click the Tenants tenant name.
3 From the Work pane, click the Faults tab.
• Fabric Faults—From the menu bar:
1 Click Fabric > Inventory.
2 From the Navigation pane, click on a Pod
3 From the Work pane, click the Faults tab.
A list of faults appears in a summary table.
Step 2
Double-click on a fault.
The fabric and system tables change to display faults that match the fault code of the fault you clicked on.
a) From the fabric or system faults, double-click on a fault in the summary table to view more information.
The Fault Properties dialog appears displaying the following tabs:
• General—Displays the following:
◦Properties—Contains information found in the summary table
◦Details—Contains fault information found in the summary table, the number of occurences, the change set,
and the original, previous, and highest severity level for the chosen fault.
• Troubleshooting—Displays the following:
◦Troubleshooting—Contains troubleshooting information that includes an explanation of the fault and the
recommended action.
◦Audit log—A tool that enables you to view the history of user-initiated events before the fault occurred. The
history is displayed in a list by a specified number of minutes. You can adjust the number of minutes by
clicking the drop-down arrow.
Cisco APIC Troubleshooting Guide
38
Using the Cisco APIC Troubleshooting Tools
Enabling Port Tracking for Uplink Failure Detection
• History—Displays history information of the affected object
Enabling Port Tracking for Uplink Failure Detection
This section explains how to enable port tracking using the GUI, NX-OS CLI, and the REST API.
Port Tracking Policy for Uplink Failure Detection
Uplink failure detection can be enabled in the fabric access global port tracking policy. The port tracking
policy monitors the status of links between leaf switches and spine switches. When an enabled port tracking
policy is triggered, the leaf switches take down all access interfaces on the switch that have EPGs deployed
on them.
Note
In the advanced GUI, port tracking is located under Fabric > Access Policies > Global Policies > Port
Tracking.
In the basic GUI, port tracking is located under System > System Settings > Port Tracking.
Depending on the model of leaf switch, each leaf switch can have 6, 8, or 12 uplink connections to each spine
switch. The port tracking policy specifies the number of uplink connections that trigger the policy, and a delay
timer for bringing the leaf switch access ports back up after the number of specified uplinks is exceeded.
The following example illustrates how a port tracking policy behaves:
• The leaf switches each have 6 active uplink connections to the spine switches.
• The port tracking policy specifies that the threshold of active uplink connections each leaf switch that
triggers the policy is 2.
• The port tracking policy triggers when the number of active uplink connections from the leaf switch to
the spine switches drops to 2.
• Each leaf switch monitors its uplink connections and triggers the port tracking policy according to the
threshold specified in the policy.
• When the uplink connections come back up, the leaf switch waits for the delay timer to expire before
bringing its access ports back up. This gives the fabric time to reconverge before allowing traffic to
resume on leaf switch access ports. Large fabrics may need the delay timer to be set for a longer time.
Note
Use caution when configuring this policy. If the port tracking setting for the number of active spine links
that triggers port tracking is too high, all leaf switch access ports will be brought down.
Cisco APIC Troubleshooting Guide
39
Using the Cisco APIC Troubleshooting Tools
Port Tracking Using the GUI
Port Tracking Using the GUI
This procedure explains how to use the Port Tracking feature using the GUI.
Step 1
From the Fabric menu, select Access Policies.
Step 2
In the left navigation pane of the Access Policies screen, select Global Polices.
Step 3
Under Global Policies tab, select the Port Tracking tab.
Step 4
Turn on the Port Tracking feature by selecting on next to Port tracking state under Properties.
Step 5
Turn off the Port Tracking feature by selecting off next to Port tracking state under Properties.
Step 6
Set the Delay restore timer, which is the configuration parameter used to specify the number of seconds before restoring
and bringing up the downlinks after the fabric port tracking starts.
Step 7
Enter the maximum number of links (any configuration value from 0 - 12) that are left up before you trigger this
configuration. (The default is zero.)
Click Submit to push your desired Port Tracking configuration to all switches on the fabric.
Step 8
Port Tracking Using the NX-OS CLI
This procedure explains how to use the Port Tracking feature using the NX-OS CLI.
Step 1
Turn on the Port Tracking feature as follows:
Example:
apic1# show porttrack
Configuration
Admin State
Bringup Delay(s)
Bringdown # Fabric Links up
Step 2
: on
: 120
: 0
Turn off the Port Tracking feature as follows:
Example:
apic1# show porttrack
Configuration
Admin State
Bringup Delay(s)
Bringdown # Fabric Links up
Cisco APIC Troubleshooting Guide
40
: off
: 120
: 0
Using the Cisco APIC Troubleshooting Tools
Port Tracking Using the REST API
Port Tracking Using the REST API
Before You Begin
This procedure explains how to use the Port Tracking feature using the REST API.
Step 1
Turn on the Port Tracking feature using the REST API as follows (admin state: on):
<polUni>
<infraInfra dn="uni/infra">
<infraPortTrackPol name="default" delay="5" minlinks="4" adminSt="on">
</infraPortTrackPol>
</infraInfra>
</polUni>
Step 2
Turn off the Port Tracking feature using the REST API as follows (admin state: off):
<polUni>
<infraInfra dn="uni/infra">
<infraPortTrackPol name="default" delay="5" minlinks="4" adminSt=“off">
</infraPortTrackPol>
</infraInfra>
</polUni>
Configuring SNMP for Monitoring and Managing Devices
This section explains how to configure SNMP using the GUI.
About SNMP
The Cisco Application Centric Infrastructure (ACI) provides extensive SNMPv1, v2, and v3 support, including
Management Information Bases (MIBs) and notifications (traps). The SNMP standard allows any third-party
applications that support the different MIBs to manage and monitor the ACI fabric.
SNMPv3 provides extended security. Each SNMPv3 device can be selectively enabled or disabled for SNMP
service. In addition, each device can be configured with a method of handling SNMPv1 and v2 requests.
For more information about using SNMP, see the Cisco ACI MIB Quick Reference.
SNMP Access Support in ACI
SNMP support in ACI is as follows:
• SNMP read queries (Get, Next, Bulk, Walk) are supported by leaf and spine switches and by APIC.
• SNMP write commands (Set) are not supported by leaf and spine switches or by APIC.
• SNMP traps (v1, v2c, and v3) are supported by leaf and spine switches and by APIC.
Cisco APIC Troubleshooting Guide
41
Using the Cisco APIC Troubleshooting Tools
Configuring the SNMP Policy Using the GUI
Note
ACI supports a maximum of 10 trap receivers.
• SNMPv3 is supported by leaf and spine switches and by APIC.
Table 1: SNMP Support Changes by Cisco APIC Release
Release
Description
1.2(2)
IPv6 support is added for SNMP trap destinations.
1.2(1)
SNMP support for the APIC controller is added. Previous releases support SNMP
only for leaf and spine switches.
For the complete list of MIBs supported in ACI, see http://www.cisco.com/c/en/us/td/docs/switches/datacenter/
aci/apic/sw/1-x/mib/list/mib-support.html.
Configuring the SNMP Policy Using the GUI
This procedure configures and enables the SNMP policy on ACI switches.
Before You Begin
To allow SNMP communications, you must configure the following:
• Configure an out-of-band contract allowing SNMP traffic. SNMP traffic typically uses UDP port 161
for SNMP requests.
• Configure the APIC out-of-band IP addresses in the 'mgmt' tenant. Although the out-of-band addresses
are configured during APIC setup, the addresses must be explicitly configured in the 'mgmt' tenant
before the out-of-band contract will take effect.
Step 1
Step 2
Step 3
In the menu bar, click Fabric.
In the submenu bar, click Fabric Policies.
In the Navigation pane, expand Pod Policies.
Step 4
Step 5
Under Pod Policies, expand Policies.
Right-click SNMP and choose Create SNMP Policy.
As an alternative to creating a new SNMP policy, you can edit the default policy fields in the same manner as described
in the following steps.
Step 6
In the SNMP policy dialog box, perform the following actions:
a) In the Name field, enter an SNMP policy name.
b) In the Admin State field, select Enabled.
c) In the Community Policies table, click the + icon, enter a Name (do not include the @ symbol) and click Update.
Cisco APIC Troubleshooting Guide
42
Using the Cisco APIC Troubleshooting Tools
Configuring an SNMP Trap Destination Using the GUI
d) (Optional) In the SNMP v3 Users table, click the + icon, enter a Name, enter the user's authentication data, and
click Update.
This step is needed only if SNMPv3 access is required.
Step 7
To configure allowed SNMP management stations, perform the following actions in the SNMP policy dialog box:
a) In the Client Group Policies table, click the + icon to open the Create SNMP Client Group Profile dialog box.
b) In the Name field, enter an SNMP client group profile name.
c) From the Associated Management EPG drop-down list, choose the management EPG.
d) In the Client Entries table, click the + icon.
e) Enter a client's name in the Name field, enter the client's IP address in the Address field, and click Update.
Note
When an SNMP management station connects with APIC using SNMPv3, APIC does not enforce the client
IP address specified in the SNMP client group profile. For SNMPv3, the management station must exist in
the Client Entries list, but the IP address need not match, as the SNMPv3 credentials alone are sufficient
for access.
Step 8
Step 9
Step 10
Click OK.
Click Submit.
Under Pod Policies, expand Policy Groups and choose a policy group or right-click Policy Groups and choose Create
POD Policy Group.
You can create a new pod policy group or you can use an existing group. The pod policy group can contain other pod
policies in addition to the SNMP policy.
Step 11
In the pod policy group dialog box, perform the following actions:
a) In the Name field, enter a pod policy group name.
b) From the SNMP Policy drop-down list, choose the SNMP policy that you configured and click Submit.
Step 12
Step 13
Under Pod Policies, expand Profiles and click default.
In the Work pane, from the Fabric Policy Group drop-down list, choose the pod policy group that you created.
Step 14
Step 15
Click Submit.
Click OK.
Configuring an SNMP Trap Destination Using the GUI
This procedure configures the host information for an SNMP manager that will receive SNMP trap notifications.
Cisco APIC Troubleshooting Guide
43
Using the Cisco APIC Troubleshooting Tools
Configuring an SNMP Trap Source Using the GUI
Note
ACI supports a maximum of 10 trap receivers. If you configure more than 10, some will not receive
notifications.
Step 1
Step 2
Step 3
In the menu bar, click Admin.
In the submenu bar, click External Data Collectors.
In the Navigation pane, expand Monitoring Destinations.
Step 4
Step 5
Right-click SNMP and choose Create SNMP Monitoring Destination Group.
In the Create SNMP Monitoring Destination Group dialog box, perform the following actions:
a) In the Name field, enter an SNMP destination name and click Next.
b) In the Create Destinations table, click the + icon to open the Create SNMP Trap Destination dialog box.
c) In the Host Name/IP field, enter an IP address or a fully qualified domain name for the destination host.
Note
Cisco APIC Release 1.2(2) and later releases support IPv6 SNMP trap destinations.
d) Choose the Port number and SNMP Version for the destination.
e) For SNMP v1 or v2c destinations, enter one of the configured community names as the Security Name and choose
noauth as v3 Security Level.
SNMP community names cannot contain the @ symbol.
f) For SNMP v3 destinations, enter one of the configured SNMP v3 user names as Security Name and choose the
desired v3 Security Level.
g) From the Management EPG drop-down list, choose the management EPG.
h) Click OK.
i) Click Finish.
Configuring an SNMP Trap Source Using the GUI
This procedure selects and enables a source object within the fabric to generate SNMP trap notifications.
Step 1
Step 2
Step 3
Step 5
In the menu bar, click Fabric.
In the submenu bar, click Fabric Policies.
In the Navigation pane, expand Monitoring Policies.
You can create an SNMP source in the Common Policy, the default policy, or you can create a new monitoring policy.
Expand the desired monitoring policy and choose Callhome/SNMP/Syslog.
If you chose the Common Policy, right-click Common Policy, choose Create SNMP Source, and follow the instructions
below for that dialog box.
In the Work pane, from the Monitoring Object drop-down list, choose ALL.
Step 6
Step 7
Step 8
From the Source Type drop-down list, choose SNMP.
In the table, click the + icon to open the Create SNMP Source dialog box.
In the Create SNMP Source dialog box, perform the following actions:
Step 4
Cisco APIC Troubleshooting Guide
44
Using the Cisco APIC Troubleshooting Tools
Monitoring the System Using SNMP
a) In the Name field, enter an SNMP policy name.
b) From the Dest Group drop-down list, choose an existing destination for sending notifications or choose Create
SNMP Monitoring Destination Group to create a new destination.
The steps for creating an SNMP destination group are described in a separate procedure.
c) Click Submit.
Monitoring the System Using SNMP
You can remotely monitor individual hosts (APIC or another host) and find out the state of any particular
node.
You can check the system's CPU and memory usage using SNMP to find out if the CPU is spiking or not.
The SNMP, a network management system, uses an SNMP client and accesses information over the APIC
and retrieves information back from it.
You can remotely access the system to figure out if the information is in the context of the network management
system and you can learn whether or not it is taking too much CPU or memory, or if there are any system or
performance issues. Once you learn the source of the issue, you can check the system health and verify whether
or not it is using too much memory or CPU.
Refer to the Cisco ACI MIB Quick Reference Manual for additional information.
Configuring SPAN for Traffic Monitoring
This section lists the SPAN guidelines and restrictions and explains how to configure SPAN sessions.
About SPAN
You can use the Switched Port Analyzer (SPAN) utility to perform detailed troubleshooting or to take a sample
of traffic from a particular application host for proactive monitoring and analysis.
SPAN copies traffic from one or more ports, VLANs, or endpoint groups (EPGs) and sends the copied traffic
to one or more destinations for analysis by a network analyzer. The process is nondisruptive to any connected
devices and is facilitated in the hardware, which prevents any unnecessary CPU load.
You can configure SPAN sessions to monitor traffic received by the source (ingress traffic), traffic transmitted
from the source (egress traffic), or both. By default, SPAN monitors all traffic, but you can configure filters
to monitor only selected traffic.
Multinode SPAN
APIC traffic monitoring policies can SPAN policies at the appropriate places to track members of each
application group and where they are connected. If any member moves, APIC automatically pushes the policy
to the new leaf switch. For example, when an endpoint VMotions to a new leaf switch, the SPAN configuration
automatically adjusts.
Cisco APIC Troubleshooting Guide
45
Using the Cisco APIC Troubleshooting Tools
SPAN Guidelines and Restrictions
SPAN Guidelines and Restrictions
• Use SPAN for troubleshooting. SPAN traffic competes with user traffic for switch resources. To minimize
the load, configure SPAN to copy only the specific traffic that you want to analyze.
• You cannot specify an l3extLIfP layer 3 subinterface as a SPAN source. You must use the entire port
for monitoring traffic from external sources.
• Tenant and access SPAN use the encapsulated remote extension of SPAN (ERSPAN) type I, while fabric
SPAN uses ERSPAN type II. For information regarding ERSPAN headers, refer to the IETF Internet
Draft at this URL: https://tools.ietf.org/html/draft-foschiano-erspan-00.
• ERSPAN destination IPs must be learned in the fabric as an endpoint.
• SPAN supports IPv6 traffic but the destination IP for the ERSPAN cannot be an IPv6 address.
• See the Verified Scalability Guide for Cisco ACI document for SPAN-related limits, such as the maximum
number of active SPAN sessions.
Configuring a SPAN Session
This procedure shows how to configure a SPAN policy to forward replicated source packets to a remote traffic
analyzer.
Step 1
Step 2
Step 3
In the menu bar, click Tenants.
In the submenu bar, click the tenant that contains the source endpoint.
In the Navigation pane, expand the tenant, expand Troubleshooting Policies, and expand SPAN.
Step 4
Under SPAN, right-click SPAN Destination Groups and choose Create SPAN Destination Group.
The Create SPAN Destination Group dialog appears.
Enter the appropriate values in the required fields of the Create SPAN Destination Group dialog box then click OK
and Submit.
Note
For a description of a field, click the information icon (i) at the top-right corner of the dialog box to display the
help file.
Under SPAN, right-click SPAN Source Groups and choose Create SPAN Source Group.
The Create SPAN Source Group dialog appears.
Enter the appropriate values in the required fields of the Create SPAN Source Group dialog box then click OK and
Submit.
Note
For a description of a field, click the information icon (i) at the top-right corner of the dialog box to display the
help file.
Step 5
Step 6
Step 7
What to Do Next
Using a traffic analyzer at the SPAN destination, you can observe the data packets from the SPAN source
EPG to verify the packet format, addresses, protocols, and other information.
Cisco APIC Troubleshooting Guide
46
Using the Cisco APIC Troubleshooting Tools
Using Statistics
Using Statistics
Statistics provide real-time measures of observed object and enable trend analysis and troubleshooting. Statistics
gathering can be configured for ongoing or on-demand collection and can be collected in cumulative counters
and gauges.
Policies define what statistics are gathered, at what intervals, and what actions to take. For example, a policy
could raise a fault on an EPG if a threshold of dropped packets on an ingress VLAN is greater than 1000 per
second.
Statistics data are gathered from a variety of sources, including interfaces, VLANs, EPGs, application
profiles,ACL rules, tenants, or internal APIC processes. Statistics accumulate data in 5-minute, 15-minute,
1-hour, 1-day, 1-week, 1-month, 1-quarter, or 1-year sampling intervals. Shorter duration intervals feed longer
intervals. A variety of statistics properties are available, including last value, cumulative, periodic, rate of
change, trend, maximum, min, average. Collection and retention times are configurable. Policies can specify
if the statistics are to be gathered from the current state of the system or to be accumulated historically or
both. For example, a policy could specify that historical statistics be gathered for 5-minute intervals over a
period of 1 hour. The 1 hour is a moving window. Once an hour has elapsed, the incoming 5 minutes of
statistics are added, and the earliest 5 minutes of data are abandoned.
Note
The maximum number of 5-minute granularity sample records is limited to 12 samples (one hour of
statistics). All other sample intervals are limited to 1,000 sample records. For example, hourly granularity
statistics can be maintained for up to 41 days.
Viewing Statistics in the GUI
You can view statistics for many objects using the APIC GUI, including application profiles, physical interfaces,
bridge domains, and fabric nodes. To view statistics in the GUI, choose the object in the navigation pane and
click the STATS tab.
Follow these steps to view statistics for an interface:
Step 1
Step 2
Step 3
Step 4
Step 5
On the menu bar, choose FABRIC > INVENTORY.
In the Navigation pane, choose a pod.
Expand the pod, and expand a switch.
In the Navigation pane, expand Interfaces and choose eth1/1.
In the Work pane, choose the STATS tab.
The APIC displays interface statistics.
What to Do Next
You can use the following icons in the Work pane to manage how the APIC displays statistics:
• Refresh—Manually refreshes statistics.
Cisco APIC Troubleshooting Guide
47
Using the Cisco APIC Troubleshooting Tools
Switch Statistics Commands
• Show Table View—Toggles between table and chart views.
• Start or Stop Stats—Enables or disables automatic refresh for statistics.
• Select Stats—Specifies the counters and sample interval to display.
• Download Object as XML—Downloads the object in XML format.
• Measurement Type (Gear icon)—Specifies the statistics measurement type. Options include cumulative,
periodic, average, or trend.
Switch Statistics Commands
You can use the following commands to display statistics on ACI leaf switches.
Command
Purpose
Legacy Cisco Nexus show/clear commands
For more information, see Cisco Nexus 9000 Series
NX-OS Configuration Guides.
show platform internal counters port [port_num | Displays spine port statistics
detail | nz | {internal [nz | int_port_num]}]
• port_num—Front port number without the slot.
• detail—Returns SNMP, class and forwarding
statistics.
• nz—Displays only non-zero values.
• internal—Displays internal port statistics.
• int_port_num—Internal logical port number.
For example, for BCM-0/97, enter 97.
Note
If there is a link reset, the counters will be
zeroed out on the switch. The conditions of
counter reset include the following:
• accidental link reset
• manually enabled port (after port is
disabled)
show platform internal counters vlan [hw_vlan_id] Displays VLAN statistics.
show platform internal counters tep [tunnel_id]
Displays TEP statistics.
show platform internal counters flow [rule_id |
{dump [asic inst] | [slice direction | index
hw_index]}]
Displays flow statistics.
clear platform internal counters port [port_num | Clears port statistics.
{internal [ int_port_num]}]
clear platform internal counters vlan [hw_vlan_id] Clears VLAN counters.
Cisco APIC Troubleshooting Guide
48
Using the Cisco APIC Troubleshooting Tools
Managing Statistics Thresholds Using the GUI
Command
Purpose
debug platform internal stats logging level
log_level
Sets the debug logging level.
debug platform internal stats logging
{err|trace|flow}
Sets the debug logging type.
Managing Statistics Thresholds Using the GUI
Step 1
Step 6
Step 7
On the menu bar, choose Fabric > Fabric Policies.
In the Navigation pane, click + to expand Monitoring Policies.
In the Navigation pane, expand the monitoring policy name (such as Default).
Click Stats Collection Policies.
In the Stats Collection Policies window, choose a Monitoring Object and Stats Type for which to set a threshold
value..
In the Work pane, Click the + icon below CONFIG THRESHOLDS.
In the THRESHOLDS FOR COLLECTION window, click + to add a threshold.
Step 8
Step 9
In the Choose a Property window, choose a statistics type.
In the EDIT STATS THRESHOLD window, specify the following threshold values:
Step 2
Step 3
Step 4
Step 5
• Normal Value—A valid value of the counter.
• Threshold Direction—Indicates whether the threshold is a maximum or minimum value.
• Rising Thresholds (Critical, Major, Minor, Warning)—Triggered when the value exceeds the threshold.
• Falling Thresholds (Critical, Major, Minor, Warning)—Triggered when the value drops below the threshold.
Step 10
Step 11
Step 12
You can specify a set and reset value for rising and falling thresholds. The set value specifies when a fault is triggered;
the reset value specifies when the fault is cleared.
Click SUBMIT to save the threshold value.
In the THRESHOLDS FOR COLLECTION window, click CLOSE.
Statistics Troubleshooting Scenarios
The following table summarizes common statistics troubleshooting scenarios for the Cisco APIC.
Cisco APIC Troubleshooting Guide
49
Using the Cisco APIC Troubleshooting Tools
Statistics Troubleshooting Scenarios
Problem
Solution
The APIC does not
enforce a configured
monitoring policy
The problem occurs when a monitoring policy is in place but the APIC does not
perform a corresponding action, such as collecting the statistics or acting on a
trigger threshold. Follow these steps to resolve the issue:
• Verify that monPolDn points to the correct monitoring policy.
• Ensure that the selectors are configured correctly and that there are no faults.
• For Tenant objects, check the relation to the monitoring policy.
Some configured statistics Follow these steps to resolve the issue:
are missing.
• Review the statistics that are disabled by default within the monitoring
policy and collection policy.
• Review the collection policy to determine if the statistics are disabled by
default or disabled for certain intervals.
• Review the statistics policy to determine if the statistics are disabled by
default or disabled for certain intervals.
Note
Statistics or history are
not maintained for the
configured time period.
Except for fabric health statistics, 5 minute statistics are stored on the
switch and are lost when the switch reboots.
Follow these steps to resolve the issue:
• Review the collection settings; if configured at the top level of the
monitoring policy, the statistics can be overridden for a specific object or
statistics type.
• Review the collection policy assigned to the monitoring object. Confirm
that the policy is present and review the administrative state, and history
retention values.
• Verify that the statistics type is configured correctly.
Some statistics are not
maintained for the full
configured interval.
Review whether the configuration exceeds the maximum historical record size.
The limitations are as follows:
• Switch statistics for 5 minute granularity are limited to 12 samples (1 hour
of 5 minute granular statistics).
• There is a hard limit of 1000 samples. For example, hourly granular statistics
can be maintained for up to 41 days.
Cisco APIC Troubleshooting Guide
50
Using the Cisco APIC Troubleshooting Tools
Statistics Cleanup
Problem
Solution
An export policy is
Follow these steps to resolve the issue:
configured but the APIC
• Check the status object for the destination policy.
does not export statistics.
• On the node that is expected to export the statistics check the export status
object and look at the export status and details properties. Aggregated EPG
stats are exported every 15 minutes from APIC nodes. Other statistics are
exported from source nodes every 5 minutes. For example, if an EPG is
deployed to two leaf switches and configured to exporte EPG aggregation
parts, then those parts are exported from the nodes every 5 minutes.
• Review whether the configuration exceeds the maximum number of export
policies. The maximum number of statistics export policies is approximately
equal to the number of tenants.
Note
5 Minute Statistics
Fluctuate
Each tenant can have multiple statistics export policies and multiple
tenants can share the same export policy, but the total number
number of policies is limited to approximately the number of
tenants.
The APIC system reports statistics every 5 minutes, sampled approximately every
10 seconds. The number of samples taken in 5 minutes may vary, because there
are slight time variances when the data is collected. As a result, the statistics
might represent a slightly longer or shorter time period. This is expected behavior.
Some historical statistics For more information, see Statistics Cleanup.
are missing.
Statistics Cleanup
The APIC and switches clean up statistics as follows:
• Switch—The switch cleans up statistics as follows:
◦5 minute statistics on switches are purged if no counter value is reported for 5 minutes. This situation
can occur when an object is deleted or statistics are disabled by a policy.
◦Statistics of larger granularity are purged if statistics are missing for more than one hour, which
can occur when:
◦Statistics are disabled by a policy.
◦A switch is disconnected from an APIC for more than one hour.
◦The switch cleans up statistics for deleted objects after 5 minutes. If an object is recreated within
this time, statistics counts remain unchanged.
◦Disabled object statistics are deleted after 5 minutes.
◦If the system state changes so that statistics reporting is disabled for 5 minutes, this switch cleans
up statistics.
Cisco APIC Troubleshooting Guide
51
Using the Cisco APIC Troubleshooting Tools
Specifying Syslog Sources and Destinations
• APIC—The APIC cleans up objects including interfaces, EPGs, temperature sensors, and health statistics
after one hour.
Specifying Syslog Sources and Destinations
This section explains how to create syslog destination groups, a syslog source, and how to enable syslog to
display in NX-OS CLI format using the REST API.
About Syslog
During operation, a fault or event in the Cisco Application Centric Infrastructure (ACI) system can trigger
the sending of a system log (syslog) message to the console, to a local file, and to a logging server on another
system. A system log message typically contains a subset of information about the fault or event. A system
log message can also contain audit log and session log entries.
Note
For a list of syslog messages that the APIC and the fabric nodes can generate, see http://www.cisco.com/
c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/syslog/guide/aci_syslog/ACI_SysMsg.html.
Many system log messages are specific to the action that a user is performing or the object that a user is
configuring or administering. These messages can be the following:
• Informational messages, providing assistance and tips about the action being performed
• Warning messages, providing information about system errors related to an object, such as a user account
or service profile, that the user is configuring or administering
In order to receive and monitor system log messages, you must specify a syslog destination, which can be the
console, a local file, or one or more remote hosts running a syslog server. In addition, you can specify the
minimum severity level of messages to be displayed on the console or captured by the file or host. The local
file for receiving syslog messages is /var/log/external/messages.
A syslog source can be any object for which an object monitoring policy can be applied. You can specify the
minimum severity level of messages to be sent, the items to be included in the syslog messages, and the syslog
destination.
You can change the display format for the Syslogs to NX-OS style format.
Additional details about the faults or events that generate these system messages are described in the Cisco
APIC Faults, Events, and System Messages Management Guide, and system log messages are listed in the
Cisco ACI System Messages Reference Guide.
Note
Not all system log messages indicate problems with your system. Some messages are purely informational,
while others may help diagnose problems with communications lines, internal hardware, or the system
software.
Cisco APIC Troubleshooting Guide
52
Using the Cisco APIC Troubleshooting Tools
Creating a Syslog Destination and Destination Group
Creating a Syslog Destination and Destination Group
This procedure configures syslog data destinations for logging and evaluation. You can export syslog data to
the console, to a local file, or to one or more syslog servers in a destination group.
Step 1
Step 2
Step 3
In the menu bar, click Admin.
In the submenu bar, click External Data Collectors.
In the Navigation pane, expand Monitoring Destinations.
Step 4
Step 5
Right-click Syslog and choose Create Syslog Monitoring Destination Group.
In the Create Syslog Monitoring Destination Group dialog box, perform the following actions:
a) In the group and profile Name field, enter a name for the monitoring destination group and profile.
b) In the group and profile Admin State drop-down list, choose enabled.
c) To enable sending of syslog messages to a local file, choose enabled from the Local File Destination Admin State
drop-down list and choose a minimum severity from the Local File Destination Severity drop-down list.
The local file for receiving syslog messages is /var/log/external/messages.
Step 6
Step 7
Step 8
d) To enable sending of syslog messages to the console, choose enabled from the Console Destination Admin State
drop-down list and choose a minimum severity from the Console Destination Severity drop-down list.
e) Click Next.
f) In the Create Remote Destinations area, click + to add a remote destination.
Caution
Risk of hostname resolution failure for remote Syslog destinations, if the DNS server used is configured to
be reachable over in-band connectivity. To avoid the issue, configure the Syslog server using the IP address,
or if you use a hostname, ensure that the DNS server is reachable over an out-of-band interface.
In the Create Syslog Remote Destination dialog box, perform the following actions:
a) In the Host field, enter an IP address or a fully qualified domain name for the destination host.
b) (Optional) In the Name field, enter a name for the destination host.
c) In the Admin State field, click the enabled radio button.
d) (Optional) Choose a minimum severity Severity, a Port number, and a syslog Forwarding Facility.
e) From the Management EPG drop-down list, choose the management endpoint group.
f) Click OK.
(Optional) To add more remote destinations to the remote destination group, click + again and repeat the steps in the
Create Syslog Remote Destination dialog box
Click Finish.
Creating a Syslog Source
A syslog source can be any object for which an object monitoring policy can be applied.
Cisco APIC Troubleshooting Guide
53
Using the Cisco APIC Troubleshooting Tools
Creating a Syslog Source
Before You Begin
Create a syslog monitoring destination group.
Step 1
From the menu bar and the navigation frame, navigate to a Monitoring Policies menu for the area of interest.
You can configure monitoring policies for tenants, fabric, and access.
Step 2
Expand Monitoring Policies, then select and expand a monitoring policy.
Under Fabric > Fabric Policies > Monitoring Policies > Common Policy is a basic monitoring policy that applies to
all faults and events and is automatically deployed to all nodes and controllers in the fabric. Alternatively, you can specify
an existing policy with a more limited scope.
Step 3
Step 4
Under the monitoring policy, click Callhome/SNMP/Syslog.
In the Work pane, choose Syslog from the Source Type drop-down list.
Step 5
From the Monitoring Object list, choose a managed object to be monitored.
If the desired object does not appear in the list, follow these steps:
a)
b)
c)
d)
Step 6
Click the Edit icon to the right of the Monitoring Object drop-down list.
From the Select Monitoring Package drop-down list, choose an object class package.
Select the checkbox for each object that you want to monitor.
Click Submit.
In a tenant monitoring policy, if you select a specific object instead of All, a Scope selection appears.
In the Scope field, select a radio button to specify the system log messages to send for this object:
• all—Send all events and faults related to this object
• specific event—Send only the specified event related to this object. From the Event drop-down list, choose the
event policy.
• specific fault—Send only the specified fault related to this object. From the Fault drop-down list, choose the fault
policy.
Step 7
Step 8
Click + to create a syslog source.
In the Create Syslog Source dialog box, perform the following actions:
a) In the Name field, enter a name for the syslog source.
b) From the Min Severity drop-down list, choose the minimum severity of system log messages to be sent.
c) In the Include field, check the checkboxes for the type of messages to be sent.
d) From the Dest Group drop-down list, choose the syslog destination group to which the system log messages will be
sent.
e) Click Submit.
Step 9
(Optional) To add more syslog sources, click + again and repeat the steps in the Create Syslog Source dialog box
Cisco APIC Troubleshooting Guide
54
Using the Cisco APIC Troubleshooting Tools
Enabling Syslog to Display in NX-OS CLI Format, Using the REST API
Enabling Syslog to Display in NX-OS CLI Format, Using the REST API
By default the Syslog format is RFC 5424 compliant. You can change the default display of Syslogs to NX-OS
type format, similar to the following example:
apic1# moquery -c "syslogRemoteDest"
Total Objects shown: 1
# syslog.RemoteDest
host
:
adminState
:
childAction
:
descr
:
dn
:
epgDn
:
format
:
forwardingFacility :
ip
:
lcOwn
:
modTs
:
monPolDn
:
name
:
operState
:
port
:
rn
:
severity
:
status
:
uid
:
vrfId
:
vrfName
:
172.23.49.77
enabled
uni/fabric/slgroup-syslog-mpod/rdst-172.23.49.77
nxos
local7
local
2016-05-17T16:51:57.231-07:00
uni/fabric/monfab-default
syslog-dest
unknown
514
rdst-172.23.49.77
information
15374
0
To enable the Syslogs to display in NX-OS type format, perform the following steps, using the REST API.
Step 1
Enable the Syslogs to display in NX-OS type format, as in the following example:
POST https://192.168.20.123/api/node/mo/uni/fabric.xml
<syslogGroup name="DestGrp77" format="nxos">
<syslogRemoteDest name="slRmtDest77" host="172.31.138.20" severity="debugging"/>
</syslogGroup>
The syslogGroup is the Syslog monitoring destination group, the sysLogRemoteDest is the name you previously configured
for your Syslog server, and the host is the IP address for the previously configured Syslog server.
Step 2
Set the Syslog format back to the default RFC 5424 format, as in the following example:
POST https://192.168.20.123/api/node/mo/uni/fabric.xml
<syslogGroup name="DestGrp77" format="aci">
<syslogRemoteDest name="slRmtDest77" host="172.31.138.20" severity="debugging"/>
</syslogGroup>
Discovering Paths and Testing Connectivity with Traceroute
This section lists the traceroute guidelines and restriction and explains how to perform a traceroute between
endpoints.
Cisco APIC Troubleshooting Guide
55
Using the Cisco APIC Troubleshooting Tools
About Traceroute
About Traceroute
The traceroute tool is used to discover the routes that packets actually take when traveling to their destination.
Traceroute identifies the path taken on a hop-by-hop basis and includes a time stamp at each hop in both
directions. You can use traceroute to test the connectivity of ports along the path between the generating
device and the device closest to the destination. If the destination cannot be reached, the path discovery traces
the path up to the point of failure.
A traceroute that is initiated from the tenant endpoints shows the default gateway as an intermediate hop that
appears at the ingress leaf switch.
Traceroute supports a variety of modes, including endpoint-to-endpoint, and leaf-to-leaf (tunnel endpoint, or
TEP to TEP). Traceroute discovers all paths across the fabric, discovers point of exits for external endpoints,
and helps to detect if any path is blocked.
Traceroute Guidelines and Restrictions
• When the traceroute source or destination is an endpoint, the endpoint must be dynamic and not static.
Unlike a dynamic endpoint (fv:CEp), a static endpoint (fv:StCEp) does not have a child object
(fv:RsCEpToPathEp) that is required for traceroute.
• Traceroute works for IPv6 source and destinations but configuring source and destination IP addresses
across IPv4 and IPv6 addresses is not allowed.
• See the Verified Scalability Guide for Cisco ACI document for traceroute-related limits.
Performing a Traceroute Between Endpoints
Step 1
Step 2
Step 3
In the menu bar, click Tenants.
In the submenu bar, click the tenant that contains the source endpoint.
In the Navigation pane, expand the tenant and expand Troubleshoot Policies.
Step 4
Under Troubleshoot Policies, right-click Endpoint-to-Endpoint Traceroute Policies and choose Create
Endpoint-to-Endpoint Traceroute Policy.
Enter the appropriate values in the Create Endpoint-to-Endpoint Traceroute Policy dialog box fields and click Submit.
Note
For the description of a field, click the information icon (i) in the top-right corner of the Create
Endpoint-to-Endpoint Traceroute Policy dialog box.
In the Navigation pane or the Traceroute Policies table, click the traceroute policy.
The traceroute policy is displayed in the Work pane.
Step 5
Step 6
Step 7
In the Work pane, click the Operational tab, click the Source End Points tab, and click the Results tab.
Step 8
In the Traceroute Results table, verify the path or paths that were used in the trace.
Note
• More than one path might have been traversed from the source node to the destination
node.
• For readability, you can increase the width of one or more columns, such as the Name
column.
Cisco APIC Troubleshooting Guide
56
Using the Cisco APIC Troubleshooting Tools
Using the Troubleshooting Wizard
Using the Troubleshooting Wizard
The Troubleshooting Wizard allows you understand and visualize how your network is behaving, which can
ease your networking concerns should issues arise.
This wizard allows you (the Administrative user) to troubleshoot issues that occur during specific time frames,
which can be designated by selecting two endpoints. For example, you may have two endpoints that are having
intermittent packet loss but you don't understand why. Through the troubleshooting GUI, you can evaluate
the issue so that you can effectively resolve it rather than logging onto each machine that you suspect to be
causing this faulty behavior.
Since you may want to revisit the session later, you should give the session a unique name. You may also
choose to use a pre-configured test. You can debug from endpoint to endpoint, or from an internal or external
endpoint, or from an external to an internal endpoint.
Further, you can define a time window in which you want to perform the debug. The Troubleshooting GUI
allows you to enter a source and destination endpoint for the endpoints you are looking for. You can do this
with a MAC, IPv4, or IPv6 address and then select by tenant. You also have the option to generate a
troubleshooting report that can be sent to TAC.
The following section describes the topology of the Troubleshooting Wizard, which is a simplified view of
the fabric with only the elements that are relevant to your two endpoints under inspection.
Note
For a list of Troubleshooting Wizard CLI commands, see the Cisco APIC Command-Line Interface User
Guide.
Getting Started with the Troubleshooting Wizard
Before you start using the Troubleshooting Wizard, you must be logged on as an Administrative user. Then
you must designate Source and Destination endpoints (Eps) and select a time window for your troubleshooting
session. The time window is used for retrieving Events, Fault Records, Deployment Records, Audit Logs,
and Statistics. (The description and time window can only be edited in the first page of the wizard, before
clicking on Start.)
Cisco APIC Troubleshooting Guide
57
Using the Cisco APIC Troubleshooting Tools
Getting Started with the Troubleshooting Wizard
Note
• You cannot modify the Source and Destination endpoints once you have clicked either the
GENERATE REPORT button or the START button. If you want to change the Source and
Destination information after you have entered it, you have to start a new session.
• As you navigate through the screens of the Troubleshooting Wizard, you have the option to take a
screen shot at any time and send it to a printer (or save it as a PDF) by clicking the Print icon (
at the top, right side of the screen. There are also Zoom In and Zoom Out icons (
you can use to modify your view of any screen.
)
) that
To set up your troubleshooting session information:
Step 1
Select OPERATIONS from the top of the screen then choose VISIBILITY & TROUBLESHOOTING.
The Visibility & Troubleshooting screen appears.
Step 2
You can choose to either use an existing troubleshooting session (using the drop-down menu) or you can create a new
one. To create a new one, enter a name for it in the Session Name field.
Step 3
Enter a description in the Description field to provide additional information.
(This step is optional.)
From the Source pull-down menu, enter a MAC, IPv4, or IPv6 address or choose an existing one.
Step 4
Step 5
Click SEARCH.
A box appears (shown as follows) displaying one or multiple rows with detailed information to help you make a selection.
Each row shows that the IP address (in the IP column) you entered is in a specific endpoint group (in the EPG column),
which belongs to a certain application (in the Application column), which is in a particular tenant (in the Tenant column).
The leaf number, fex number, and port details are shown in the Learned At column.
Step 6
From the Destination pull-down menu, enter a MAC, IPv4, or IPv6 address or choose an existing one.
Step 7
Click SEARCH.
A box appears displaying one or multiple rows to help you make a selection (as previously described for the Source
endpoint search).
Check the External IP checkbox if you are using an endpoint to external internet protocol.
Note
• For more information about endpoints and external IPs, refer to the Cisco Application Centric Infrastructure
Fundamentals guide.
Step 8
• Ideally, you should select the Source and Destination endpoints from the same tenant or some of the
troubleshooting functionality may be impacted, as explained later in this document. Once you make
selections for these endpoints, you can learn about the topology that connects the two in the Faults
troubleshooting screen.
Step 9
Select a time window by making selections from the From (for session Start time) and To (for session End time) pull-down
menus.
The Time Window (shown as follows) is used for debugging an issue that occurred during a specific time frame in the
past, and is used for retrieving Events, All Records, Deployment Records, Audit Logs, and Statistics. There are two sets
of windows; one for all records and one for individual leafs (or nodes).
Cisco APIC Troubleshooting Guide
58
Using the Cisco APIC Troubleshooting Tools
Generating Troubleshooting Reports
Note
You have two options for setting the time window, which you can toggle back and forth from using the Use
fixed time checkbox.
• You can specify a rolling time window based on any number of Latest Minutes (the default is 240 minutes
but this can be changed).
• Or, you can specify a fixed time window for the session in the From and To fields by checking the Use
fixed time checkbox.
The default time window is based on a default of latest 240 minutes (which means that the session contains
data for the past 240 minutes) preceding the time you created the session. You can also set up or modify time
window information from the bottom of the left navigation pane.
Click START at the bottom right side of the screen to begin your troubleshooting session.
The topology diagram for your troubleshooting session loads and then appears.
Note
For a list of Troubleshooting Wizard CLI commands, see the Cisco APIC Command-Line Interface User Guide.
Note
Step 10
Generating Troubleshooting Reports
You can generate a troubleshooting report in several formats, including JSON, XML, PDF, and HTML. Once
you select a format, you can download the report (or schedule a download of the report) and use it for offline
analysis or you can send it to TAC so that a support case can be created.
To generate a troubleshooting report:
Step 1
From the bottom right corner of the screen, click GENERATE REPORT.
The Generate Report dialog box appears.
Step 2
Choose an output format from the Report Format drop-down menu (XML, HTML, JSON, or PDF).
Step 3
If you want to schedule the download of the report to happen immediately, click theNow > SUBMIT.
An Information box appears indicating where to obtain the report once it has been generated.
Step 4
To schedule the generation of the report for a later time, choose a schedule by clicking Use a scheduler > Scheduler
drop-down menu then choose either an existing schedule or create a new one by clicking Create Scheduler. .
The CREATE TRIGGER SCHEDULE dialog appears.
Step 5
Enter information for the Name, Description (optional), and Schedule Windows fields.
Note
For more information on how to use the SCHEDULER, please refer to the online
help.
Click SUBMIT.
Step 6
The reports take some time to generate (from a couple of minutes to up to ten minutes), depending on the size
of the fabric and how many faults or events exist. A status message displays while the report is being generated.
To retrieve and view the troubleshooting report, click SHOW GENERATED REPORTS.
Supply the credentials (User Name and Password) of the server in the Authentication Required window.
The troubleshooting report is then downloaded locally to your system.
Cisco APIC Troubleshooting Guide
59
Using the Cisco APIC Troubleshooting Tools
Topology in the Troubleshooting Wizard
The ALL REPORTS window appears showing a list of all the reports that have been generated, including
the one you just triggered. From there, you can click the link to either download or immediately view the
report, depending on the output file format you chose (for example, if the file is a PDF, it may open immediately
in your browser).
Topology in the Troubleshooting Wizard
This section explains the topology in the Troubleshooting Wizard. The topology shows how the Source and
Destination end points (Eps) are connected to the fabric, what the network path is from the Source to the
Destination, and what the intermediate switches are.
The Source end point is displayed on the left side of the topology and the Destination end point is on the right,
as shown in the following wizard topology diagram.
Note
This wizard topology only shows the leafs, spines, and fexes of the devices involved in the traffic from
the Source end point to the Destination end point. However, there may be many other leafs (tens or hundreds
of leafs and many other spines) that exist.
This topology also shows links, ports, and devices. If you hover over the
icon), you can see the tenant
that the Ep belongs to, which application it belongs to, and the traffic encapsulation it is using (such as VLAN).
There is a color legend on the left side of the screen (shown as follows) that describes the severity levels
associated with each color in the topology diagram (for example, critical versus minor).
Hovering over items such as boxes or ports in the topology provides more detailed information. If the port or
link has a color, this means that there is a problem for you to troubleshoot. For example, if the color is red or
Cisco APIC Troubleshooting Guide
60
Using the Cisco APIC Troubleshooting Tools
Using the Faults Troubleshooting Screen
orange, this indicates that there is a fault on a port or link. If the color is white, then there are no faults that
exist. If the link has a number in a circle, it indicates how many parallel links between the same two nodes
are affected by a fault of the severity given by the color of the circle. Hovering over a port allows you to see
which port is connected to the Source Ep.
Right-clicking on a leaf allows you to access the console of the switch. A pop-up window appears that allows
you to log into that device.
Note
• If there are L4 though L7 services (firewall and load balancer) they will be shown in the topology
as well
• For a topology with the load balancer, the destination is expected to be the VIP (Virtual IP)
• When the endpoint is behind an ESX server, the ESX is shown in the topology
Using the Faults Troubleshooting Screen
Click Faults in the Navigation pane to begin using the Faults troubleshooting screen.
The Faults screen shows the topology that connects the two endpoints that you previously selected as well
as the faults that were found. Only faults for the designated communication are shown. Wherever there are
faults, they are highlighted in a certain color to convey the severity. Refer to the color legend at the top of the
screen (shown as follows) to understand the severity levels associated with each color. This topology also
shows the relevant leafs, spines, and fexes to your troubleshooting session. Hovering over items such as leafs,
spines, and fexes (or clicking on faults) provides more detailed information for analysis.
Cisco APIC Troubleshooting Guide
61
Using the Cisco APIC Troubleshooting Tools
Using the Drop/Statistics Troubleshooting Screen
Note
White boxes indicate that there are no issues to troubleshoot in that particular area.
Clicking on a fault displays a dialog box with two tabs (FAULTS and RECORDS) that contain more detailed
information for analysis, including Severity, Affected Object, Creation Time, Last Transaction, Lifecycle,
and Description fields.
Related Topics
Using the Drop/Statistics Troubleshooting Screen, on page 62
Using the Drop/Statistics Troubleshooting Screen
Click Drop/Stats in the Navigation pane to begin using the Drop/Stats troubleshooting screen.
The Drop/Stats window displays the topology with all the statistics from the drops so that you can clearly
see where drops exist or not. You can click on any drop image to see more information for analysis.
Once you click a drop image, there are three tabs at the top of the Drop/Stats screen, and the statistics shown
are localized to that particular leaf or switch.
The three statistics tabs are:
• DROP STATS
Cisco APIC Troubleshooting Guide
62
Using the Cisco APIC Troubleshooting Tools
Using the Contracts Troubleshooting Screen
This tab shows the statistics for drop counters. The packets that are dropped at various levels are shown
here.
Note
By default, counters with zero values are hidden but the user can choose to see all the
values.
• CONTRACT DROPS
This tab shows a list of the contract drops that have occurred, which are individual packet logs (ACL
logs), and shows information for each packet such as the Source Interface, Source IP address, Source
Port, Destination IP address, Destination Port, and Protocol.
Note
Not every packet is displayed here.
• TRAFFIC STATS
This tab shows the statistics that indicate ongoing traffic. These are how many packets have been
transferring.
Note
By default, counters with zero values are hidden but the user can choose to see all the
values.
You can also view all of the statistics for all managed objects at once by clicking the All icon (
in the top, left corner of the screen.
) located
You also have the option to pick zero or non-zero drops. Checking the box for Show stats with zero values
(located in the top, left corner of the screen) allows you to see all the existing drops. The fields for Time,
Affected Object, Stats, and Value become populated with data for all the zero values.
If you do not check the Show stats with zero values box, you will see results with non-zero drops.
Note
The same logic applies if you click the All icon. All three tabs (DROP STATS, CONTRACT DROPS,
and TRAFFIC STATS) are also available and have the same type of information appearing.
Related Topics
Using the Contracts Troubleshooting Screen, on page 63
Using the Contracts Troubleshooting Screen
Click Contracts in the Navigation pane to begin using the Contracts troubleshooting screen.
The Contracts troubleshooting screen displays the contracts that are applicable from the Source to the
Destination and from the Destination to the Source.
Cisco APIC Troubleshooting Guide
63
Using the Cisco APIC Troubleshooting Tools
Using the Events Troubleshooting Screen
Each one of the blue table heading rows indicates a filter. There are multiple rows under each filter that indicate
multiple filter entries (Protocol, L4 Src, L4 Dest, TCP Flags, Action, Nodes, and Hits) for a particular leaf
or switch.
Hovering over the certificate icon, shows you the contract name and the contract filter name. The text appearing
on the right side of each blue table heading row (or filter) tells what type of contract it is, for example:
• Epg to Epg
• BD Allow
• Any to Any
• Context Deny
These contracts are categorized from the Source to the Destination and from the Destination to the Source.
Note
The hits shown for each filter are cumulative (that is, the total hits for that contract hit, contract filter, or
rule are shown for each particular leaf.) Statistics are refreshed automatically every (one) minute.
You can get policy information by hovering over the Information (
are being referred to.
Note
) icon. You can also see which EPGs
If there are no contracts between the endpoints, this will be indicated with a There is no contract data
pop-up.
Related Topics
Using the Events Troubleshooting Screen, on page 64
Using the Events Troubleshooting Screen
Click Events and Audits in the Navigation pane to begin using the Events and Audits troubleshooting
screen.
If you click on an individual leaf or spine switch, you can see more detailed information about that individual
event.
There are two tabs available: EVENTS and DEPLOYMENT RECORDS.
• EVENTS show event records for any changes that have occurred in systems (such as physical interfaces
or VLANS, for example). There are individual events listed for each particular leaf. You can sort these
events based on Severity, Affected Object, Creation Time, Cause, and Description.
• DEPLOYMENT RECORDS show the deployment of policies on physical interfaces, VLANs, VXLANs,
and L3 CTXs. These records show the time when a VLAN was placed on a leaf because of the epg.
If you click the All icon (
) for the All Changes screen, you can see all the events indicating any changes
that have occurred during your specified time interval (or troubleshooting session).
There are three tabs in the All Changes screen, including:
Cisco APIC Troubleshooting Guide
64
Using the Cisco APIC Troubleshooting Tools
Using the Traceroute Troubleshooting Screen
• AUDITS
Audits do not have a leaf association, which is why they are only available in the All Changes screen.
• EVENTS (described above)
• DEPLOYMENT RECORDS (described above)
Related Topics
Using the Traceroute Troubleshooting Screen, on page 65
Using the Traceroute Troubleshooting Screen
Click Traceroute in the Navigation pane to begin using the Traceroute troubleshooting screen.
To create and run a traceroute for troubleshooting:
1 In the TRACEROUTE dialog box, choose a destination port from the Destination Port drop-down menu.
2 Choose a protocol from the Protocol pull-down menu. The options supported include:
• icmp —This protocol is uni-directional, in that it does a traceroute from the Source leaf to the
Destination endpoint only.
• tcp—This protocol is also bi-directional, as described above for the udp protocol.
• udp—This protocol is bi-directional, in that it does a traceroute from the Source leaf to the Destination
endpoint, then from the Destination leaf back to the Source endpoint.
Note
UDP, TCP and ICMP are the only supported protocols for IPv4. For IPv6, only UDP is supported.
3 Once you create a traceroute, click the Play (or Start) button to start the traceroute.
Note
When you press the Play button, the polices are created on the system and a Warning message appears.
4 ClickOK to proceed and the traceroute starts to run.
5 Click the Stop button to end the traceroute.
Note
When you press the Stop button, the policies are removed from the system.
Once the traceroute completes, you can see where it was launched and what the result was. There is a pull-down
menu next to Traceroute Results that shows where the traceroute was launched (from the Source to the
Destination or from the Destination to the Source).
The result is also shown in the Traceroute dialog, which includes information for Running Time, Traceroute
Status, Destination Port, and Protocol.
The results are represented by green and/or red arrows. A green arrow is used to represent each node in the
path that responded to the traceroute probes. The beginning of a red arrow represents where the path ends as
Cisco APIC Troubleshooting Guide
65
Using the Cisco APIC Troubleshooting Tools
Using the Atomic Counter Troubleshooting Screen
that's the last node that responded to the traceroute probes. You don't choose which direction to launch the
traceroute. Instead, the traceroute is always started for the session. If the session is:
• EP to external IP or external IP to EP, the traceroute is always launched from EP to external IP.
• EP to EP and protocol is ICMP, the traceroute is always launched from the source to the destination.
• EP to EP and protocol is UDP/TCP, the traceroute is always bidirectional.
Note
• The Traceroute Results drop-down menu can be used to expose/visualize the results for each
direction for scenario #3 above. In scenarios #1 and #2, it's always greyed out.
• If the Traceroute Status shows as incomplete, this means you are still waiting for part of the data
to come back. If the Traceroute Status shows as complete, then it is actually complete.
Related Topics
Using the Atomic Counter Troubleshooting Screen, on page 66
Using the Atomic Counter Troubleshooting Screen
Click Atomic Counter in the Navigation pane to begin using the Atomic Counter troubleshooting screen.
The Atomic Counter screen is used to take source and destination information and create a counter policy
based on that. You can create an atomic counter policy between two endpoints and monitor the traffic going
back and forth from the Source to the Destination and from the Destination to the Source. You can determine
how much traffic is going through and especially determine if any anomalies (drops or excess packets) are
reported between the source and destination leaves.
There are Play (or Start) and Stop buttons at the top of the screen so that you can start and stop the atomic
counter policy at any point and can count the packets that are being sent.
Note
When you press the Play button, the polices are created on the system and the packet counter starts. When
you press the Stop button, the policies are removed from the system.
The results are shown in two different formats. You can view them in either a brief format, which includes a
summary, or in a longer format (by clicking on the Expand button). Both brief and expanded formats show
both directions. The expanded format shows the cumulative counts plus the counts for each of the latest 30s
intervals, while the brief format only shows the counts for cumulative and last interval.
Related Topics
Using the SPAN Troubleshooting Screen, on page 66
Using the SPAN Troubleshooting Screen
Click SPAN in the Navigation pane to begin using the SPAN troubleshooting screen.
Using this screen, you can span (or mirror) bi-directional traffic and redirect it to the analyzer. In a SPAN
session, you are making a copy and sending it to the analyzer.
Cisco APIC Troubleshooting Guide
66
Using the Cisco APIC Troubleshooting Tools
L4 - L7 Services Validated Scenarios
This copy goes to a particular host (the analyzer IP address) and then you can use a software tool such as
Wireshark to view the packets. The session information has source and destination information, session type,
and the timestamp range.
Note
When you press the Play button, the polices are created on the system. When you press the Stop button,
the policies are removed from the system.
Note
For a list of Troubleshooting Wizard CLI commands, see the Cisco APIC Command-Line Interface User
Guide.
L4 - L7 Services Validated Scenarios
The Troubleshooting Wizard allows you to provide two endpoints and see the corresponding topology between
those endpoints. When L4 - L7 services exist between the two endpoints in the topology, you are able to view
these as well.
This section describes the L4 - L7 scenarios that have been validated for this release. Within the L4 - L7
services, the number of topologies is very high, which means that you can have different configurations for
firewalls, load balancers, and combinations of each. If a firewall exists between the two endpoints in the
topology, the Troubleshooting Wizard retrieves the firewall data and connectivity from the firewall to the
leafs. If a load balancer exists between the two endpoints, you can retrieve and view information up to the
load balancer but not up to the server.
The following table shows the L4 - L7 service scenarios that were validated for the Troubleshooting Wizard:
Scenario
1
2
3
4
5
6
Number of
Nodes
1
1
2
1
1
2
Device
GoTo FW
(vrf split)
GoTo
SLB
GoTo,GoTo
FW,SLB
FW-GoThrough SLB-GoTo
FW, SLB
(GoThrough,
GoTo)
2
2
2
2
2
Number of Arms 2
Consumer
EPG
EPG
EPG
L3Out
L3Out
L3Out
Provider
EPG
EPG
EPG
EPG
EPG
EPG
Device Type
VM
VM
VM
physical
physical
physical
Contract Scope
tenant
context
context
context
context
global
L2
L2, L2
L3, L2
L3
L3 / L2,L3
BSW
DL/PC
regular port
vPC
regular port
Connector Mode L2
Service Attach
BSW
Cisco APIC Troubleshooting Guide
67
Using the Cisco APIC Troubleshooting Tools
List of APIs for Endpoint to Endpoint Connections
Scenario
1
2
3
4
5
Client Attach
FEX
FEX
FEX
Regular Port Regular
Port
Server Attach
vPC
vPC
vPC
regular port
6
regular port
regular port regular port
List of APIs for Endpoint to Endpoint Connections
The following is a list of the available Troubleshooting Wizard APIs for EP to EP (endpoint to endpoint)
connections:
• interactive API, on page 68
• createsession API, on page 69
• modifysession API, on page 71
• atomiccounter API, on page 71
• traceroute API, on page 71
• span API, on page 72
• generatereport API, on page 73
• schedulereport API, on page 74
• getreportstatus API, on page 74
• getreportslist API, on page 75
• getsessionslist API, on page 75
• getsessiondetail API, on page 75
• deletesession API, on page 75
• clearreports API, on page 76
• contracts API, on page 77
interactive API
To create an endpoint (ep) to endpoint interactive troubleshooting session, use the interactive API. The
module name is troubleshoot.eptoeputils.topo and the function is getTopo. The required argument (req_args)
for the interactive API is - session.
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
Cisco APIC Troubleshooting Guide
68
Using the Cisco APIC Troubleshooting Tools
createsession API
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- description
Description about the session
- scheduler
Scheduler name for report generation
- srcepid
Obsolete
- dstepid
Obsolete
- include
Obsolete
- format
Format of report to be generated
- ui
Used internally (ignore)
- sessionurl
Location of the report
-action
Start/stop/status etc. for traceroute/atomiccounter
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
createsession API
To create an endpoint (ep) to endpoint troubleshooting session, use the createsession API. The module name
is troubleshoot.eptoeputils.session and the function is createSession.
Cisco APIC Troubleshooting Guide
69
Using the Cisco APIC Troubleshooting Tools
createsession API
The required argument (req_args) for the createsession API is - session (session name).
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- description
Description about the session
- format
Format of report to be generated
- ui
Used internally (ignore)
-action
Start/stop/status etc. for traceroute/atomiccounter
- scheduler
- srctenant
Name of the tenant for the source endpoint
- srcapp
Name of the app for the source endpoint
- srcepg
Name of the endpoint group for the source endpoint
- dsttenant
Name of the tenant for the destination endpoint
- dstapp
Name of the app for the destination endpoint
- dstepg
Name of the endpoint group for the destination endpoint
- mode
Used internally
Cisco APIC Troubleshooting Guide
70
Using the Cisco APIC Troubleshooting Tools
modifysession API
modifysession API
To modify an endpoint (ep) to endpoint troubleshooting session, use the modifysession API. The module
name is troubleshoot.eptoeputils.topoand the function is modifySession.
The required arguments (req_args) for the modifysession API are - session (session name) and - mode.
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- description
Description about the session
atomiccounter API
To create an endpoint (ep) to endpoint atomic counter session, use the atomiccounter API. The module name
is troubleshoot.eptoeputils.atomiccounter and the function is manageAtomicCounterPols.
The required arguments (req_args) for the atomiccounter API include:
• - session
• - action
• - mode
Note
There are no optional arguments (opt_args) for the atomiccounter API.
traceroute API
To create an endpoint (ep) to endpoint traceroute session using the API, use the traceroute API. The module
name is troubleshoot.eptoeputils.traceroute and the function is manageTraceroutePols.
The required arguments (req_args) for the traceroute API include:
• - session (session name)
• - action (start/stop/status)
Cisco APIC Troubleshooting Guide
71
Using the Cisco APIC Troubleshooting Tools
span API
• - mode
Syntax Description
Optional Arguments (opt_args)
Description
- protocol
Protocol name
- dstport
Destination port name
span API
To create an endpoint (ep) to endpoint span troubleshooting session, use the span API. The module name is
troubleshoot.eptoeputils.span and the function is monitor.
The required arguments (req_args) for the span API include:
• - session (session name)
• - action (start/stop/status)
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting
from start time (in minutes)
- description
Description about the session
- scheduler
Scheduler name for report generation
Cisco APIC Troubleshooting Guide
72
Using the Cisco APIC Troubleshooting Tools
generatereport API
- srcepid
Obsolete
- dstepid
Obsolete
- include
Obsolete
- format
Format of report to be generated
- ui
Used internally (ignore)
- sessionurl
Location of the report
-action
Start/stop/status etc. for traceroute/atomiccounter
- srctenant
Name of the tenant for the source endpoint
- srcapp
Name of the app for the source endpoint
- srcepg
Name of the endpoint group for the source endpoint
- dsttenant
Name of the tenant for the destination endpoint
- dstapp
Name of the app for the destination endpoint
- dstepg
Name of the endpoint group for the destination endpoint
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
generatereport API
To generate a troubleshooting report using the API, use the generatereport API. The module name is
troubleshoot.eptoeputils.report and the function is generateReport.
The required arguments (req_args) for the generatereport API are - session (session name) and - mode.
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- include
Obsolete
- format
Format of report to be generated
Cisco APIC Troubleshooting Guide
73
Using the Cisco APIC Troubleshooting Tools
schedulereport API
schedulereport API
To schedule the generation of a troubleshooting report using the API, use the schedulereport API. The module
name is troubleshoot.eptoeputils.report and the function is scheduleReport. The required argument
(req_args) for the schedulereport API is - session
The required arguments (req_args) for the schedulereport API include:
• - session (session name)
• - scheduler (scheduler name)
• - mode
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- include
Obsolete
- format
Format of report to be generated
- action
Start/stop/status etc. for traceroute/atomiccounter
getreportstatus API
To get the status of a generated report using the API, use the getreportstatus API. The module name is
troubleshoot.eptoeputils.report and the function is getStatus.
The required arguments (req_args) for the getreportstatus API include:
• - session (session name)
• - sessionurl (session URL)
• - mode
Note
There are no optional arguments (opt_args) for the getreportstatus API.
Cisco APIC Troubleshooting Guide
74
Using the Cisco APIC Troubleshooting Tools
getreportslist API
getreportslist API
To get a list of generated reports using the API, use the getreportslist API. The module name is
troubleshoot.eptoeputils.report and the function is getReportsList.
The required arguments (req_args) for the getreportslist API are- session (session name) and - mode.
Note
There are no optional arguments (opt_args) for the getreportslist API.
getsessionslist API
To get a list of troubleshooting sessions using the API, use the getsessionslist API. The module name is
troubleshoot.eptoeputils.session and the function is getSessions.
The required argument (req_args) for the getsessionlist API is - mode.
Note
There are no optional arguments (opt_args) for the getsessionlist API.
getsessiondetail API
To get specific details about a troubleshooting session using the API, use the getsessiondetail API. The module
name is troubleshoot.eptoeputils.session and the function is getSessionDetail.
The required arguments (req_args) for the getsessiondetail API are - session (session name) and - mode.
Note
There are no optional arguments (opt_args) for the getsessiondetail API.
deletesession API
To delete a particular troubleshooting session using the API, use the deletesession API. The module name is
troubleshoot.eptoeputils.session and the function is deleteSession.
The required argument (req_args) for the deletesession API is - session (session name).
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
Cisco APIC Troubleshooting Guide
75
Using the Cisco APIC Troubleshooting Tools
clearreports API
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- description
Description about the session
- scheduler
Scheduler name for report generation
- srcepid
Obsolete
- dstepid
Obsolete
- include
Obsolete
- format
Format of report to be generated
- ui
Used internally (ignore)
- sessionurl
Location of report
- action
Start/stop/status etc. for traceroute/atomiccounter
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
clearreports API
To clear the list of generated reports using the API, use the clearreports API. The module name is
troubleshoot.eptoeputils.report and the function is clearReports.
The required arguments (req_args) for the clearreports API are- session (session name) and - mode.
Cisco APIC Troubleshooting Guide
76
Using the Cisco APIC Troubleshooting Tools
contracts API
Note
There are no optional arguments (opt_args) for the clearreports API.
contracts API
To get contracts information using the API, use the contracts API. The module name is
troubleshoot.eptoeputils.contracts and the function is getContracts.
The required arguments (req_args) for the contracts API are- session (session name) and -mode.
There are no optional arguments (opt_args) for the contracts API.
List of APIs for Endpoint to Layer 3 External Connections
The following is a list of the available Troubleshooting Wizard APIs for EP to EP (endpoint to endpoint)
connections:
• interactive API, on page 78
• modifysession API, on page 79
• atomiccounter API, on page 80
• traceroute API, on page 81
• span API, on page 82
• generatereport API, on page 83
• schedulereport API, on page 84
• getreportstatus API, on page 74
• getreportslist API, on page 75
• clearreports API, on page 76
• createsession API, on page 78
• getsessionslist API, on page 85
• getsessiondetail API, on page 87
• deletesession API, on page 88
• contracts API, on page 88
• ratelimit API, on page 89
• 13ext API, on page 90
Cisco APIC Troubleshooting Guide
77
Using the Cisco APIC Troubleshooting Tools
interactive API
interactive API
To create an endpoint (ep) to Layer 3 (L3) external interactive troubleshooting session, use the interactive
API. The module name is troubleshoot.epextutils.epext_topo and the function is getTopo. The required
arguments (req_args) for the interactive API are - session, - include, and - mode.
The following table shows the optional argument (opt_args):
Syntax Description
Optional Arguments (opt_args)
Description
- refresh
createsession API
To create an endpoint (Ep) to Layer 3 (L3) external troubleshooting session using the API, use the createsession
API. The module name is troubleshoot.epextutils.epextsession and the function is createSession. The
required argument (req_args) for the createsession API is - session (session name).
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- description
Description about the session
Cisco APIC Troubleshooting Guide
78
Using the Cisco APIC Troubleshooting Tools
modifysession API
- scheduler
Scheduler name for report generation
- srcepid
Obsolete
- dstepid
Obsolete
- include
Obsolete
- format
Format of report to be generated
- ui
Used internally (ignore)
- sessionurl
Location of the report
-action
Start/stop/status etc. for traceroute/atomiccounter
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
modifysession API
To modify an endpoint (Ep) to Layer 3 (L3) external troubleshooting session, use the modifysession API.
The module name is troubleshoot.epextutils.epextsessionand the function is modifySession. The required
argument (req_args) for the modifysession API is - session (session name).
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
Cisco APIC Troubleshooting Guide
79
Using the Cisco APIC Troubleshooting Tools
atomiccounter API
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- description
Description about the session
- scheduler
Scheduler name for report generation
- srcepid
Obsolete
- dstepid
Obsolete
- include
Obsolete
- format
Format of report to be generated
- ui
Used internally (ignore)
- sessionurl
Location of the report
-action
Start/stop/status etc. for traceroute/atomiccounter
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
atomiccounter API
To create an endpoint (ep) to endpoint atomic counter session, use the atomiccounter API. The module name
is troubleshoot.epextutils.epext_ac and the function is manageAtomicCounterPols.
The required arguments (req_args) for the atomiccounter API include:
• - session (session name)
• - action (start/stop/status)
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
Cisco APIC Troubleshooting Guide
80
Using the Cisco APIC Troubleshooting Tools
traceroute API
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- ui
Used internally (ignore)
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
traceroute API
To create an endpoint (ep) to to Layer 3 external traceroute troubleshooting session using the API, use the
traceroute API. The module name is troubleshoot.epextutils.epext_traceroute and the function is
manageTraceroutePols.
The required arguments (req_args) for the traceroute API include:
• - session (session name)
• - action (start/stop/status)
Syntax Description
Optional Arguments (opt_args)
Description
- protocol
Protocol name
- dstport
Destination port name
- srcep
Source endpoint
- dstep
Destination endpoint
- srcip
Source IP address
Cisco APIC Troubleshooting Guide
81
Using the Cisco APIC Troubleshooting Tools
span API
- dstip
Destination IP address
- srcextip
Source external IP address
- dstIp
Destination external IP address
- ui
Used internally (ignore)
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
span API
To create an endpoint (Ep) to Layer 3 (L3) external span troubleshooting session, use the span API. The
module name is troubleshoot.epextutils.epext_span and the function is monitor.
The required arguments (req_args) for the span API include:
• - session (session name)
• - action (start/stop/status)
• - mode
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- portslist
List of ports
- dstapic
Destination APIC
- srcipprefix
Source endpoint IP address prefix
- flowid
Flow ID
- dstepg
Destination endpoint group
- dstip
Destination endpoint IP address
- analyser
???
- desttype
Destination type
- spansrcports
Span source ports
Cisco APIC Troubleshooting Guide
82
Using the Cisco APIC Troubleshooting Tools
generatereport API
generatereport API
To generate a troubleshooting report using the API, use the generatereport API. The module name is
troubleshoot.eptoeputils.report and the function is generateReport.
The required argument (req_args) for the generatereport API is - session (session name).
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- description
Description about the session
- scheduler
Scheduler name for report generation
- srcepid
Obsolete
- dstepid
Obsolete
- include
Obsolete
- format
Format of report to be generated
- ui
Used internally (ignore)
- sessionurl
Location of the report
-action
Start/stop/status etc. for traceroute/atomiccounter
Cisco APIC Troubleshooting Guide
83
Using the Cisco APIC Troubleshooting Tools
schedulereport API
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
schedulereport API
To schedule the generation of a troubleshooting report using the API, use the schedulereport API. The module
name is troubleshoot.eptoeputils.report and the function is scheduleReport. The required argument
(req_args) for the schedulereport API is - session
The required arguments (req_args) for the schedulereport API include:
• - session (session name)
• - scheduler (scheduler name)
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint
- dstep
Destination endpoint
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- description
Description about the session
- srcepid
Obsolete
- dstepid
Obsolete
Cisco APIC Troubleshooting Guide
84
Using the Cisco APIC Troubleshooting Tools
getreportstatus API
- include
Obsolete
- format
Format of report to be generated
- ui
Used internally (ignore)
- sessionurl
Location of the report
-action
Start/stop/status etc. for traceroute/atomiccounter
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
getreportstatus API
To get the status of a generated report using the API, use the getreportstatus API. The module name is
troubleshoot.eptoeputils.report and the function is getStatus.
The required arguments (req_args) for the getreportstatus API include:
• - session (session name)
• - sessionurl (session URL)
• - mode
Note
There are no optional arguments (opt_args) for the getreportstatus API.
getreportslist API
To get a list of generated reports using the API, use the getreportslist API. The module name is
troubleshoot.eptoeputils.report and the function is getReportsList.
The required arguments (req_args) for the getreportslist API are- session (session name) and - mode.
Note
There are no optional arguments (opt_args) for the getreportslist API.
getsessionslist API
To get a list of troubleshooting sessions using the API, use the getsessionslist API. The module name is
troubleshoot.epextutils.epextsession and the function is getSessions.
Cisco APIC Troubleshooting Guide
85
Using the Cisco APIC Troubleshooting Tools
getsessionslist API
Note
There are no required arguments for this API.
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- session
Session name
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- description
Description about the session
- scheduler
Scheduler name for report generation
- srcepid
Obsolete
- dstepid
Obsolete
- include
Obsolete
- format
Format of report to be generated
- ui
Used internally (ignore)
- sessionurl
Location of report
- action
Start/stop/status etc. for traceroute/atomiccounter
- mode
Used internally
Cisco APIC Troubleshooting Guide
86
Using the Cisco APIC Troubleshooting Tools
getsessiondetail API
- _dc
Used internally
- ctx
Used internally
getsessiondetail API
To get specific details about a troubleshooting session using the API, use the getsessiondetail API. The module
name is troubleshoot.epextutils.session and the function is getSessionDetail. The required argument
(req_args) for the getsessiondetail API is - session (session name).
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- description
Description about the session
- scheduler
Scheduler name for report generation
- srcepid
Obsolete
- dstepid
Obsolete
- include
Obsolete
- format
Format of report to be generated
Cisco APIC Troubleshooting Guide
87
Using the Cisco APIC Troubleshooting Tools
deletesession API
- ui
Used internally (ignore)
- sessionurl
Location of report
- action
Start/stop/status etc. for traceroute/atomiccounter
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
deletesession API
To delete a particular troubleshooting session using the API, use the deletesession API. The module name is
troubleshoot.epextutils.epextsession and the function is deleteSession.
The required arguments (req_args) for the deletesession API are - session (session name) and - mode.
Note
There are no optional arguments (opt_args) for the deletesession API.
clearreports API
To clear the list of generated reports using the API, use the clearreports API. The module name is
troubleshoot.eptoeputils.report and the function is clearReports.
The required arguments (req_args) for the clearreports API are- session (session name) and - mode.
Note
There are no optional arguments (opt_args) for the clearreports API.
contracts API
To get contracts information using the API, use the contracts API. The module name is
troubleshoot.epextutils.epext_contracts and the function is getContracts. The required argument (req_args)
for the contracts API is - session (session name).
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
Cisco APIC Troubleshooting Guide
88
Using the Cisco APIC Troubleshooting Tools
ratelimit API
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- epext
Endpoint to external
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
- ui
Used internally (ignore)
ratelimit API
This section provides information on the the ratelimit API. The module name is
troubleshoot.eptoeputils.ratelimit and the function is control. The required argument (req_args) for the
ratelimit API is - action (start/stop/status).
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
Cisco APIC Troubleshooting Guide
89
Using the Cisco APIC Troubleshooting Tools
13ext API
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- epext
Endpoint to external
- mode
Used internally
- _dc
Used internally
- ctx
Used internally
13ext API
This section provides information on the the 13ext API. The module name is troubleshoot.epextutils.l3ext
and the function is execute. The required argument (req_args) for the 13ext API is - action (start/stop/status).
The following table lists the optional arguments (opt_args) and descriptions for each.
Syntax Description
Optional Arguments (opt_args)
Description
- srcep
Source endpoint name
- dstep
Destination endpoint name
- srcip
Source endpoint IP address
- dstip
Destination endpoint IP address
- srcmac
Source endpoint MAC
- dstmac
Destination endpoint MAC
- srcextip
L3 external source IP address
- dstextip
L3 external destination IP address
- starttime
Start time of the troubleshooting session
- endtime
End time of the troubleshooting session
Cisco APIC Troubleshooting Guide
90
Using the Cisco APIC Troubleshooting Tools
13ext API
- latestmin
Time window for the troubleshooting session starting from
start time (in minutes)
- epext
Endpoint to external
- mode
Used internally
Cisco APIC Troubleshooting Guide
91
Using the Cisco APIC Troubleshooting Tools
13ext API
Cisco APIC Troubleshooting Guide
92
CHAPTER
7
Manually Removing Disabled Interfaces and
Decommissioned Switches from the GUI
In a scenario where a fabric port is shut down then brought back up, it is possible that the port entry will
remain disabled in the GUI. If this occurs, no operations can be performed on the port. To resolve this, the
port must be manually removed from the GUI.
• Manually Removing Disabled Interfaces and Decommissioned Switches from the GUI, page 93
Manually Removing Disabled Interfaces and Decommissioned
Switches from the GUI
This section explains how to manually remove disabled interfaces and decommissioned switches in the GUI.
Step 1
Step 2
Step 3
From the Fabric tab, click Inventory.
In the Navigation pane, click Disabled Interfaces and Decommissioned Switches.
The list of disabled interfaces and decommissioned switches appears in a summary table in the Work pane.
From the Work pane, right-click on the interface or switch that you want to remove and choose Delete.
Cisco APIC Troubleshooting Guide
93
Manually Removing Disabled Interfaces and Decommissioned Switches from the GUI
Manually Removing Disabled Interfaces and Decommissioned Switches from the GUI
Cisco APIC Troubleshooting Guide
94
CHAPTER
8
Troubleshooting Steps for Endpoint Connectivity
Problems
This chapter lists the steps for troubleshooting endpoint connectivity issues using the Cisco APIC tools,
contains procedures for inspecting the operational status of your endpoints and tunnel interfaces, and explains
how to connect an SFP module.
This chapter contains the following sections:
• Troubleshooting Endpoint Connectivity, page 95
• Inspecting Endpoint and Tunnel Interface Status, page 96
• Connecting an SFP Module, page 97
Troubleshooting Endpoint Connectivity
Step 1
Step 2
Inspect the operational status of each endpoint.
The operational status will reveal any fault or misconfiguration of the endpoints. See
Inspecting the Endpoint Status, on page 96 .
Inspect the status of the tunnel interface.
The operational status will reveal any fault or misconfiguration of the tunnel. See Inspecting the Tunnel Interface Status,
on page 97.
Step 3
Perform a traceroute between the endpoint groups (EPGs).
A traceroute will reveal any problems with intermediate nodes, such as spine nodes, between the endpoints. See Performing
a Traceroute Between Endpoints, on page 56.
Step 4
Configure an atomic counter on an endpoint.
The atomic counter will confirm whether the source endpoint is transmitting packets or the destination endpoint is
receiving packets, and whether the number of packets received equals the number of packets sent. See Configuring
Atomic Counters, on page 30.
Step 5
Inspect the contracts under each EPG.
Cisco APIC Troubleshooting Guide
95
Troubleshooting Steps for Endpoint Connectivity Problems
Inspecting Endpoint and Tunnel Interface Status
Inspect the contracts under each EPG to make sure they allow the traffic that should flow between the EPGs. As a test,
you can temporarily open the contracts to allow unrestricted traffic.
Step 6
Configure a SPAN policy to forward source packets to a monitoring node.
A packet analyzer on the monitoring node will reveal any packet issues such as an incorrect address or protocol. See
Configuring a SPAN Session, on page 46.
Inspecting Endpoint and Tunnel Interface Status
This section explains how to inspect the operational status of endpoints and tunnel interfaces. Performing
these procedures enables you to reveal any fault or misconfiguration of the endpoints and tunnel interfaces.
Inspecting the Endpoint Status
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Step 8
Step 9
Step 10
In the menu bar, click Tenants.
In the submenu bar, click the tenant that contains the source endpoint.
In the Navigation pane, expand the tenant, expand Application Profiles, and expand the application profile that contains
the endpoint.
Expand Application EPGs and click the EPG to be inspected.
In the Work pane, from the list of endpoints in the Endpoint table, double-click the source endpoint to open the Client
End Point dialog box.
In the Client End Point dialog box, verify the endpoint properties and click the Operational tab.
In the Operational tab, view the health, status, and fault information.
In the Status table, click any items with entries, such as changes, events, or faults.
Close the Client End Point dialog box.
In the Endpoint table, view the Interface entry for the endpoint and note the node and tunnel IDs.
Repeat this procedure for the destination endpoint.
Note
Occasionally, bidirectional traffic is interrupted between IP addresses in two micro-segmented EPGs deployed
behind two leaf switches in the fabric. This can occur when the IP addresses are transitioning because of a
configuration change from micro-segment EPG to base EPG. Or conversely, this can occur on two different
leaf switches at the same time while bidirectional traffic is running. In this case, the policy tag for each remote
endpoint still points to its previous EPG.
Workaround: Manually clear the remote endpoints on the switches or wait for the remote endpoint to age out.
To clear the endpoints, log on to the CLI on each switch and enter the clear system internal epm endpoint
command with the appropriate option. For example, if your endpoints are based on the IP address, enter clear
system internal epm endpoint key vrf vrf_name{ip | ipv6} ip-address. The endpoints are then relearned with
the correct policy tag.
Cisco APIC Troubleshooting Guide
96
Troubleshooting Steps for Endpoint Connectivity Problems
Inspecting the Tunnel Interface Status
Inspecting the Tunnel Interface Status
This procedure shows how to inspect the operational status of the tunnel interface.
Step 1
Step 2
Step 3
In the menu bar, click Fabric.
In the submenu bar, click Inventory.
In the Navigation pane, expand the pod and expand the node ID of the source endpoint interface.
Step 4
Step 5
Under the node, expand Interfaces, expand Tunnel Interfaces, and click the tunnel ID of the source endpoint interface.
In the Work pane, verify the tunnel interface properties and click the Operational tab.
Step 6
In the Operational tab, view the health, status, and fault information.
In the Status table, click any items with entries, such as changes, events, or faults.
Repeat this procedure for the destination endpoint interface.
Step 7
Connecting an SFP Module
When you connect an SFP module to a new card, you need to create a link speed policy for the module to
communicate with the card. Follow these steps to create a link speed policy.
Step 1
Create an interface policy to specify the link speed:
Example:
<fabricHIfPol name=”SpeedPol” speed=”1G”/>
Step 2
Reference the link speed policy within an interface policy group:
Example:
<infraAccPortGrp name=”myGroup”>
<infraRsHIfPol tnFabricHIfPolName=”SpeedPol”/>
</infraAccPortGrp>
Cisco APIC Troubleshooting Guide
97
Troubleshooting Steps for Endpoint Connectivity Problems
Connecting an SFP Module
Cisco APIC Troubleshooting Guide
98
CHAPTER
9
Troubleshooting EVPN Type-2 Route
Advertisement
• Troubleshooting EVPN Type-2 Route Distribution to a DCIG, page 99
Troubleshooting EVPN Type-2 Route Distribution to a DCIG
For optimal traffic forwarding in an EVPN topology, you can enable fabric spines to distribute host routes to
a Data Center Interconnect Gateway (DCIG) using EVPN type-2 (MAC-IP) routes along with the public BD
subnets in the form of BGP EVPN type-5 (IP Prefix) routes. This is enabled using the HostLeak object. If
you encounter problems with route distribution, use the steps in this topic to troubleshoot.
SUMMARY STEPS
1. Verify that HostLeak object is enabled under the VRF-AF in question, by entering a command such as
the following in the spine-switch CLI:
2. Verify that the config-MO has been successfully processed by BGP, by entering a command such as the
following in the spine-switch CLI:
3. Verify that the public BD-subnet has been advertised to DCIG as an EVPN type-5 route:
4. Verify whether the host route advertised to the EVPN peer was an EVPN type-2 MAC-IP route:
5. Verify that the EVPN peer (a DCIG) received the correct type-2 MAC-IP route and the host route was
successfully imported into the given VRF, by entering a command such as the following on the DCIG
device (assuming that the DCIG is a Cisco ASR 9000 switch in the example below):
DETAILED STEPS
Step 1
Verify that HostLeak object is enabled under the VRF-AF in question, by entering a command such as the following in
the spine-switch CLI:
Cisco APIC Troubleshooting Guide
99
Troubleshooting EVPN Type-2 Route Advertisement
Troubleshooting EVPN Type-2 Route Distribution to a DCIG
Example:
spine1# ls /mit/sys/bgp/inst/dom-apple/af-ipv4-ucast/
ctrl-l2vpn-evpn ctrl-vpnv4-ucast hostleak summary
Step 2
Verify that the config-MO has been successfully processed by BGP, by entering a command such as the following in
the spine-switch CLI:
Example:
spine1# show bgp process vrf apple
Look for output similar to the following:
Information for address family IPv4 Unicast in VRF apple
Table Id
: 0
Table state
: UP
Table refcount
: 3
Peers
Active-peers
Routes
Paths
Networks
0
0
0
0
0
Aggregates
0
Redistribution
None
Wait for IGP convergence is not configured
GOLF EVPN MAC-IP route is enabled
EVPN network next-hop 192.41.1.1
EVPN network route-map map_pfxleakctrl_v4
Import route-map rtctrlmap-apple-v4
EVPN import route-map rtctrlmap-evpn-apple-v4
Step 3
Verify that the public BD-subnet has been advertised to DCIG as an EVPN type-5 route:
Example:
spine1# show bgp l2vpn evpn 10.6.0.0 vrf overlay-1
Route Distinguisher: 192.41.1.5:4123
(L3VNI 2097154)
BGP routing table entry for [5]:[0]:[0]:[16]:[10.6.0.0]:[0.0.0.0]/224, version 2088
Paths: (1 available, best #1)
Flags: (0x000002 00000000) on xmit-list, is not in rib/evpn
Multipath: eBGP iBGP
Advertised path-id 1
Path type: local 0x4000008c 0x0 ref 1, path is valid, is best path
AS-Path: NONE, path locally originated
192.41.1.1 (metric 0) from 0.0.0.0 (192.41.1.5)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 2097154
Community: 1234:444
Extcommunity:
RT:1234:5101
4BYTEAS-GENERIC:T:1234:444
Path-id 1 advertised to peers:
50.41.50.1
In the Path type entry, ref 1 indicates that one route was sent.
Step 4
Verify whether the host route advertised to the EVPN peer was an EVPN type-2 MAC-IP route:
Example:
spine1# show bgp l2vpn evpn 10.6.41.1 vrf overlay-1
Route Distinguisher: 10.10.41.2:100
(L2VNI 100)
BGP routing table entry for [2]:[0]:[2097154]:[48]:[0200.0000.0002]:[32]:[10.6.41
.1]/272, version 1146
Cisco APIC Troubleshooting Guide
100
Troubleshooting EVPN Type-2 Route Advertisement
Troubleshooting EVPN Type-2 Route Distribution to a DCIG
Shared RD: 192.41.1.5:4123
(L3VNI 2097154)
Paths: (1 available, best #1)
Flags: (0x00010a 00000000) on xmit-list, is not in rib/evpn
Multipath: eBGP iBGP
Advertised path-id 1
Path type: local 0x4000008c 0x0 ref 0, path is valid, is best path
AS-Path: NONE, path locally originated
EVPN network: [5]:[0]:[0]:[16]:[10.6.0.0]:[0.0.0.0] (VRF apple)
10.10.41.2 (metric 0) from 0.0.0.0 (192.41.1.5)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 2097154 2097154
Extcommunity:
RT:1234:16777216
Path-id 1 advertised to peers:
50.41.50.1
The Shared RD line indicates the RD/VNI shared by the EVPN type-2 route and the BD subnet.
The EVPN Network line shows the EVPN type-5 route of the BD-Subnet.
The Path-id advertised to peers indicates the path advertised to EVPN peers.
Step 5
Verify that the EVPN peer (a DCIG) received the correct type-2 MAC-IP route and the host route was successfully
imported into the given VRF, by entering a command such as the following on the DCIG device (assuming that the
DCIG is a Cisco ASR 9000 switch in the example below):
Example:
RP/0/RSP0/CPU0:asr9k#show bgp vrf apple-2887482362-8-1 10.6.41.1
Tue Sep 6 23:38:50.034 UTC
BGP routing table entry for 10.6.41.1/32, Route Distinguisher: 44.55.66.77:51
Versions:
Process
bRIB/RIB SendTblVer
Speaker
2088
2088
Last Modified: Feb 21 08:30:36.850 for 28w2d
Paths: (1 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Not advertised to any peer
Local
192.41.1.1 (metric 42) from 10.10.41.1 (192.41.1.5)
Received Label 2097154
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported
Received Path ID 0, Local Path ID 1, version 2088
Community: 1234:444
Extended community: 0x0204:1234:444 Encapsulation Type:8 Router
MAC:0200.c029.0101 RT:1234:5101
RIB RNH: table_id 0xe0000190, Encap 8, VNI 2097154, MAC Address: 0200.c029.0101,
IP Address: 192.41.1.1, IP table_id 0x00000000
Source AFI: L2VPN EVPN, Source VRF: default,
Source Route Distinguisher: 192.41.1.5:4123
In this output, the received RD, next hop, and attributes are the same for the type-2 route and the BD subnet.
Cisco APIC Troubleshooting Guide
101
Troubleshooting EVPN Type-2 Route Advertisement
Troubleshooting EVPN Type-2 Route Distribution to a DCIG
Cisco APIC Troubleshooting Guide
102
CHAPTER
10
Performing a Rebuild of the Fabric
This chapter explains how to rebuild your fabric.
• Rebuilding the Fabric, page 103
Rebuilding the Fabric
Caution
This procedure is extremely disruptive. It eliminates the existing fabric and recreates a new one.
This procedure allows you to rebuild (reinitialize) your fabric, which you may need to do for any of the
following reasons:
• To change the TEP IPs
• To change the Infra VLAN
• To change the fabric name
• To perform TAC troubleshooting tasks
Deleting the APICs erases the configuration on them and brings them up in the startup script. Performing this
on the APICs can be done in any order, but ensure that you perform the procedure on all of them (every leaf
and spine in the fabric).
Before You Begin
Ensure that the following is in place:
• Regularly scheduled backups of the configuration
• Console access to the leaves and spines
• A configured and reachable CIMC, which is necessary for KVM console access
Cisco APIC Troubleshooting Guide
103
Performing a Rebuild of the Fabric
Rebuilding the Fabric
• No Java issues
Step 1
Step 2
If you would like to retain your current configuration, you can perform a configuration export using the following
procedure: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Using_Import_Export_to_
Recover_Config_States.html
Erase the configuration on the APICs by connecting to the KVM console and entering the following commands:
a) >acidiag touch clean
b) >acidiag touch setup
c) >acidiag reboot
Ensure that each node boots up in fabric discovery mode and is not part of the previously configured fabric.
The acidiag touch command alone is not useful for this procedure, because it does not bring the APIC up in
the startup script.
Caution
It is extremely important that you ensure that all previous fabric configurations have been removed. If any
previous fabric configuration exists on even a single node, the fabric cannot be rebuilt.
When all previous configurations have been removed, run the startup script for all APICs. At this point, you can change
any of the above values, TEP, TEP Vlan, and/or Fabric Name. Ensure that these are consistent across all APICs. For
more information, refer to: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/getting-started/
b_APIC_Getting_Started_Guide/b_APIC_Getting_Started_Guide_chapter_01.html#concept_
F46E2193E3134CD090B65B16038D11A9.
To clean reboot the fabric nodes, log in to each fabric node and execute the following:
a) >setup-clean-config.sh
b) >reload
Note
Step 3
Step 4
Step 5
Step 6
Log in to apic1 and perform a configuration import using the following procedure: http://www.cisco.com/c/en/us/td/
docs/switches/datacenter/aci/apic/sw/kb/b_KB_Using_Import_Export_to_Recover_Config_States.html.
Wait for a few minutes as the fabric now uses the previous fabric registration policies to rebuild the fabric over the nodes.
(Depending on the size of the fabric, this step may take awhile.)
Cisco APIC Troubleshooting Guide
104
CHAPTER
11
Verifying IP-Based EPG Configurations
There are two types of endpoint groups (EPGs) that you can create: application EPGs and IP-based EPGs.
IP-based EPGs differ from regular application EPGs in that they are microsegment EPGs. This chapter
explains how to verify that your IP-based EPG configurations are properly classified as IP-based using the
GUI or using switch commands.
This chapter contains the following sections:
• Verifying IP-Based EPG Configurations Using the GUI, page 105
• Verifying IP-EPG Configurations Using Switch Commands, page 106
Verifying IP-Based EPG Configurations Using the GUI
This procedure explains how to verify that you have correctly configured an IP-based EPG using the GUI
and Visore tool.
Step 1
Step 2
Step 3
Step 4
Step 5
Verify that the IP-based EPG you created is listed under the uSeg EPGs folder in the GUI (shown in the following screen
capture).
Note that there is one IP-based EPG listed under uSeg EPGs named "IP" that was created using the REST API.
Verify that the information is correct in the EPG - IP properties screen (right side window pane) for each EPG IP (IP-based
EPG).
Note the list of IP-based EPGs and IP addresses that are shown at the bottom of the screen.
From your web browser, enter the APIC IP address followed by "/visore.html." Visore is a tool that allows you to view
all the objects in the system, such as EPGs. You can use Visore to verify that your IP-based EPGs have been properly
configured. For more information about Visore, see the Application Policy Infrastructure Controller Visore Tool
Introduction document.
Enter your username and password then click Login to log into Visore.
Run a query for the IP-based EPGs that you verified in the GUI by entering the name of the class in the field next to
Class or DN (for example, "fvAEPg").
Cisco APIC Troubleshooting Guide
105
Verifying IP-Based EPG Configurations
Verifying IP-EPG Configurations Using Switch Commands
This is a view from the APIC point of view. You can see that the "Total objects shown" above is "3", meaning
there are three EPGs that were downloaded to the switch. You can see that the IP-based EPG that was previously
listed in the GUI as "IP" is now shown next to "dn". Also note that "yes" is displayed next to "isAttrBasedEPg",
which means that this has been properly configured as an IP-based EPG. You can verify all the objects have
been configured successfully using Visore, including both application EPGs and IP-based EPGs.
This is a view from the switch point of view. On the switch, you can run a query for the fvEpP class to see the EPGs and
check for the "crtrnEnabled" attribute. It will be set to "yes" for IP-based EPGs.
Verify that under this EPG, the children of the EPG are shown with IP addresses to ensure a proper configuration. For
each IP address configured, there is one object (named "l3IpCktEp") that the switch uses to classify the traffic. Once the
configuration is there, when the packets arrive, the switch uses these objects to classify them.
Verify that the pcTags for all the endpoints and IP addresses that you configured match. Every EPG has a pcTag. All
the endpoints that match with the IP addresses you configured are classified into this pcTag. Every endpoint has an IP
address that you can run a class query on. When you are troubleshooting, you want to verify whether these endpoints
(servers) are properly getting classified into this IP-based EPG or not. (The pcTags should match for the IP-based EPG.)
Note
Step 6
Step 7
Verifying IP-EPG Configurations Using Switch Commands
This procedure explains how to use switch commands to verify you IP-EPG ("IpCkt") configurations.
Step 1
Step 2
Step 3
Step 4
Log in to the leaf.
Navigate to the /mit/sys directory.
In the /mit/sys directory, find ctx (vrf context directory)
In the VRF cts directory, go to the specific BD directory where the IpCkt is configured.
You should see the IpCkt.
Note
"IpCkt" and "IP-EPG" are used interchangeably in this document.
Step 5
Step 6
Step 7
Navigate to the directory and the "cat summary" gives you the information regarding IpCkt.
Ensure that the summary's "operSt' does not say "unsupported".
Find out the VLAN ID that corresponds to the BD where the IpCkt is configured.
Note
The VLAN ID can be found through any of the show vlan internal bd-info commands or through the show
system internal epm vlan all command.
Step 8
Once you find the VLAN ID of the BD, issue show system internal epm <vlan-id> detail.
Here you should be able to see all the configured IpCkts with a specific sclass. (It should match that of what you see in
the /mit/sys directory.)
Repeat the steps for vsh_lc that you followed for vsh.
Send the traffic with an IP matching the IpCtk in the BD, and through show system internal epm endp ip <a.b.c.d>,
you can verify that the learned IP has the IP-flags for "sclass" and a specific sclass value.
Repeat the steps for vsh_lc that you followed for vsh.
Step 9
Step 10
Step 11
List of the Switch Troubleshooting Commands Used in this Procedure:
Cd /mits/sys/ctx-vxlan…/bd-vxlan…
Cisco APIC Troubleshooting Guide
106
Verifying IP-Based EPG Configurations
Verifying IP-EPG Configurations Using Switch Commands
Vsh -c
Vsh -c
Vsh -c
Vsh -c
Vsh_lc
Vsh_lc
Vsh_lc
vsh_lc
vsh_lc
cat summary
“show system internal epm vlan all” or
“show vlan internal bd-info”
“show system internal epm vlan <vlan-id> detail”
“show system internal epm endp ip <a.b.c.d>"
-c “show system internal epm vlan all” or
-c “show vlan internal bd-info”
-c “show system internal epm vlan <vlan-id> detail”
-c “show system internal epm endp ip <a.b.c.d>”
-c “show system internal epm epg”
Cisco APIC Troubleshooting Guide
107
Verifying IP-Based EPG Configurations
Verifying IP-EPG Configurations Using Switch Commands
Cisco APIC Troubleshooting Guide
108
CHAPTER
12
Recovering a Disconnected Leaf
If all fabric interfaces on a leaf are disabled (interfaces connecting a leaf to the spine) due to a configuration
pushed to the leaf, connectivity to the leaf is lost forever and the leaf becomes inactive in the fabric. Trying
to push a configuration to the leaf does not work because connectivity has been lost. This chapter describes
how to recover a disconnected leaf.
• Recovering a Disconnected Leaf Using the REST API, page 109
Recovering a Disconnected Leaf Using the REST API
To recover a disconnected leaf, at least one of the fabric interfaces must be enabled using the following process.
The remaining interfaces can be enabled using the GUI, REST API, or CLI.
To enable the first interface, post a policy using the REST API to delete the policy posted and bring the fabric
ports Out-of-Service. You can post a policy to the leaf to bring the port that is Out-of-Service to In-Service
as follows:
Note
Step 1
In the following examples, the assumption is that 1/49 is one of the leaf ports connecting to the spine.
Clear the blacklist policy from the APIC (using the REST API).
Example:
$APIC_Address/api/policymgr/mo/.xml
<polUni>
<fabricInst>
<fabricOOServicePol>
<fabricRsOosPath tDn="topology/pod-1/paths-$LEAF_Id/pathep-[eth1/49]"
lc="blacklist" status ="deleted" />
</fabricOOServicePol>
</fabricInst>
</polUni>
Step 2
Post a local task to the node itself to bring up the interfaces you want using l1EthIfSetInServiceLTask.
Cisco APIC Troubleshooting Guide
109
Recovering a Disconnected Leaf
Recovering a Disconnected Leaf Using the REST API
Example:
$LEAF_Address/api/node/mo/topology/pod-1/node-$LEAF_Id/sys/action.xml
<actionLSubj oDn="sys/phys-[eth1/49]">
<l1EthIfSetInServiceLTask adminSt='start'/>
</actionLSubj>
Cisco APIC Troubleshooting Guide
110
CHAPTER
13
Determining Why a PIM Interface Was Not
Created
A PIM interface (pim:if) is created for L3Out interfaces (note that L3Out SVI interfaces are not supported),
multicast tunnel interfaces (per VRF), SVI interfaces corresponding to PIM-enabled pervasive BDs, and
loopback interfaces on border leafs (each per VRF).
This chapter contains troubleshooting information for situations where the pim:if is not being created. For
more information on PIM, see the Cisco ACI and Layer 3 Multicast with Cisco ACI and the Cisco Application
Centric Infrastructure Fundamentals guides.
This chapter contains the following sections:
• A PIM Interface Was Not Created For an L3Out Interface, page 111
• A PIM Interface Was Not Created For a Multicast Tunnel Interface, page 112
• A PIM Interface Was Not Created For a Multicast-Enabled Bridge Domain, page 112
A PIM Interface Was Not Created For an L3Out Interface
If a PIM interface (pim:If) is not being created for an L3Out interface, confirm the following:
1 PIM is enabled on the L3Out. If PIM is disabled, enable it.
2 If PIM is enabled on the container L3Out, confirm that a multicast l3ext:InstP has been created with
"__int_" as a prefixed name. This multicast l3ext:InstP is used to deploy L3Out PIM policies to the switches.
There should be one multicast l3ext:InstP per L3Out.
Note
• If a multicast l3ext:InstP exists on the IFC, we can check whether a corresponding fv:RtdEpP is
created and deployed on each switch where there is an interface in that L3Out.
• We do not support an L3Out SVI interface for PIM.
Cisco APIC Troubleshooting Guide
111
Determining Why a PIM Interface Was Not Created
A PIM Interface Was Not Created For a Multicast Tunnel Interface
A PIM Interface Was Not Created For a Multicast Tunnel
Interface
If a PIM interface (pim:if) is not created for a multicast tunnel interface (tunnel:If), confirm the following:
1 The corresponding tunnel:If has been created.
Note
The tunnel:If should have type “underlay-mcast."
2 Each mcast-enabled VRF has created an mcast tunnel.
3 The destination IP field of the tunnel:If is populated with a valid GIPO address.
4 If the tunnel:If is not populated with a valid GIPO address, check the pim:CtxP on the IFC and the
pim:CtxDef on the switches to make sure GIPO is allocated correctly.
5 The source IP of the tunnel:If has the loopback address of an L3Out for BL and “127.0.0.100” for NBL.
A PIM Interface Was Not Created For a Multicast-Enabled
Bridge Domain
If a PIM interface (pim:if) is not created for a multicast-enabled bridge domain (BD), confirm the following:
1 The corresponding BD or corresponding Ctx has PIM enabled.
2 The corresponding BD is pervasive.
3 The pervasive BD-based pim:If takes default parameters.
Note
For interaction with igmp snooping, when PIM is enabled on a pervasive BD, the routing bit should be
automatically enabled for the corresponding igmpsnoop:If.
Cisco APIC Troubleshooting Guide
112
CHAPTER
14
Confirming the Port Security Installation
This chapter explains how to confirm the port security installation in the APIC and leaf switch using Visore
and how to confirm port security has been programmed in the hardware using the Cisco NX-OS-style CLI.
For information about configuring port security, see the Cisco Port Security document.
This chapter contains the following sections:
• Confirming Your Port Security Installation Using Visore , page 113
• Confirming Your Hardware Port Security Installation Using the Cisco NX-OS CLI, page 113
Confirming Your Port Security Installation Using Visore
Step 1
Step 2
On the Cisco APIC, run a query for the l2PortSecurityPol class in Visore to verify the port security policy installation.
On the leaf switch, run a query for l2PortSecurityPolDef in Visore to confirm that the concrete object exists on the
interface.
If you have confirmed that port security is installed on the Cisco APIC and leaf switch, use the Cisco NX-OS CLI to
confirm that port security has been programmed in the hardware.
Confirming Your Hardware Port Security Installation Using the
Cisco NX-OS CLI
Step 1
View the port security status on the switch interface as follows:
Example:
switch# show system internal epm interface ethernet 1/35 det
name : Ethernet1/35 ::: if index : 0x1a022000 ::: state : UP
vPC : No ::: EPT : 0x0
Cisco APIC Troubleshooting Guide
113
Confirming the Port Security Installation
Confirming Your Hardware Port Security Installation Using the Cisco NX-OS CLI
MAC Limit : 8 ::: Learn Disable : No ::: PortSecurity Action : Protect
VLANs : 4-23
Endpoint count : 5
Active Endpoint count : 5
switch# show system internal epm interface port-channel 1 det
name : port-channel1 ::: if index : 0x16000000 ::: state : UP
vPC : No ::: EPT : 0x0
MAC Limit : 6 ::: Learn Disable : No ::: PortSecurity Action : Protect
VLANs :
Endpoint count : 0
Active Endpoint count : 0
Number of member ports : 1
Interface : Ethernet1/34
/0x1a021000
::::
Step 2
View the port security status on the module interface as follows:
Example:
module-1# show system internal epmc interface ethernet 1/35 det
if index : 0x1a022000 ::: name : Ethernet1/35 ::: tun_ip = 0.0.0.0
MAC limit : 8 ::: is_learn_disable : No ::: MAC limit action: Protect
pc if index : 0 ::: name :
is_vpc_fc FALSE ::: num_mem_ports : 0
interface state : up
Endpoint count : 5
EPT : 0
module-1# show system internal epmc interface port-channel 1 det
if index : 0x16000000 ::: name : port-channel1 ::: tun_ip = 0.0.0.0
MAC limit : 6 ::: is_learn_disable : No ::: MAC limit action: Protect
pc if index : 0 ::: name :
is_vpc_fc FALSE ::: num_mem_ports : 1
interface state : up
Endpoint count : 0
EPT : 0
::::
Step 3
View the port security status on the leaf switch as follows:
Example:
swtb15-leaf2# show system internal epm interface ethernet 1/35 det
name : Ethernet1/35 ::: if index : 0x1a022000 ::: state : UP
vPC : No ::: EPT : 0x0
MAC Limit : 5 ::: Learn Disable : Yes ::: PortSecurity Action : Protect
VLANs : 4-23
Endpoint count : 5
Active Endpoint count : 5
::::
Step 4
Confirm the MAC limit on the module interface as follows:
Example:
module-1# show system internal eltmc info interface port-channel1 | grep mac_limit
mac_limit_reached:
0
:::
mac_limit:
8
port_sec_feature_set:
1
::: mac_limit_action:
1
Example:
module-1# show system internal eltmc info interface ethernet 1/35 | grep mac_limit
mac_limit_reached:
0
:::
mac_limit:
8
port_sec_feature_set:
1
::: mac_limit_action:
1
Step 5
View the port security status in the module and confirm the MAC limit as follows:
Cisco APIC Troubleshooting Guide
114
Confirming the Port Security Installation
Confirming Your Hardware Port Security Installation Using the Cisco NX-OS CLI
Example:
module-1# show system internal epmc interface ethernet 1/35 det
if index : 0x1a022000 ::: name : Ethernet1/35 ::: tun_ip = 0.0.0.0
MAC limit : 5 ::: is_learn_disable : Yes ::: MAC limit action: Protect
pc if index : 0 ::: name :
is_vpc_fc FALSE ::: num_mem_ports : 0
interface state : up
Endpoint count : 5
EPT : 0
::::
Example:
module-1# show system internal eltmc info interface ethernet 1/35 | grep mac_limit
mac_limit_reached:
1
:::
mac_limit:
5
port_sec_feature_set:
1
::: mac_limit_action:
1
module-1# exit
Cisco APIC Troubleshooting Guide
115
Confirming the Port Security Installation
Confirming Your Hardware Port Security Installation Using the Cisco NX-OS CLI
Cisco APIC Troubleshooting Guide
116
CHAPTER
15
Troubleshooting QoS Policies
This section provides solutions for troubleshooting QoS policies.
• Troubleshooting Cisco APIC QoS Policies, page 117
Troubleshooting Cisco APIC QoS Policies
The following table summarizes common troubleshooting scenarios for the Cisco APIC QoS.
Problem
Unable to update a configured QoS policy.
Solution
1 Invoke the following API to ensure that
qospDscpRule is present on the leaf.
GET
https://192.0.20.123/api/node/class/qospDscpRule.xml
2 Ensure that the QoS rules are accurately
configured and associated to the EPG ID to which
the policy is attached.
Use the following NX-OS style CLI commands
to verify the configuration.
leaf1#
leaf1#
show vlan
show system internal aclqos qos policy
detail
show running-config tenant
tenant-name policy-map type qos
custom-qos-policy-name
apic1# show running-config tenant
tenant-name application application-name epg
epg-name
apic1#
Cisco APIC Troubleshooting Guide
117
Troubleshooting QoS Policies
Troubleshooting Cisco APIC QoS Policies
Cisco APIC Troubleshooting Guide
118
CHAPTER
16
Determining the Supported SSL Ciphers
This chapter explains how to determine which SSL ciphers are supported.
• About SSL Ciphers, page 119
• Determining the Supported SSL Ciphers Using the CLI , page 120
About SSL Ciphers
The Cisco Application Centric Infrastructure (ACI) Representational State Transfer (REST) Application
Programming Interface (API) has gone through an evolution from the day the solution debuted to recent
versions where the HTTPS/SSL/TLS support has gotten increasingly more stringent. This document is intended
to cover the evolution of HTTPS, SSL, and TLS support on the Cisco ACI REST API and provide customers
with a guide of what is required for a client to utilize the REST API securely.
HTTPS is a protocol that utilizes either Secure Socket Layers (SSL) or Transport Layer Security (TLS) to
form a secure connection for a HTTP session. SSL or TLS is used to encrypt the traffic between a client and
a HTTP server. In addition, servers that support HTTPS have a certificate that can usually be used by the
client to verify the server's authenticity. This is the opposite of the client authenticating with the server. In
this case, the server is saying, "I am server_xyz and here is the certificate that proves it." The client can then
utilize that certificate to verify the server is "server_xyz."
There are other important aspects to SSL/TLS that involve the supported encryption ciphers available in each
protocol as well as the inherent security of the SSL or TLS protocols. SSL has gone through three iterations
- SSLv1, SSLv2 and SSLv3 - all of which are now considered insecure. TLS has gone through three iterations
- TLSv1, TLSv1.1 and TLSv1.2 - of which only TLSv1.1 and TLSv1.2 are considered "secure." Ideally, a
client should utilize the highest available TLS version it can and the server should support only TLSv1.1 and
TLSv1.2. However, most servers must keep TLSv1 for outdated clients.
Almost all modern browsers support both TLSv1.1 and TLSv1.2. However, a client that utilizes HTTPS may
not be a browser. The client may be a java application or a python script that communicates with a web server
and must negotiate HTTPS/TLS. In this type of a situation, the questions of what is supported and where
becomes much more important.
Cisco APIC Troubleshooting Guide
119
Determining the Supported SSL Ciphers
Determining the Supported SSL Ciphers Using the CLI
Determining the Supported SSL Ciphers Using the CLI
Before You Begin
This section describes how to use the CLI to determine which SSL ciphers are supported.
Step 1
Get the supported ciphers in your openssl environment, shown as follows:
Example:
openssl ciphers 'ALL:eNULL'
Step 2
Separate the ciphers using sed or some other tool, shown as follows:
Example:
openssl ciphers 'ALL:eNULL' |
Step 3
sed -e 's/:/\n/g'
Loop over the ciphers and poll the APIC to see which ones are supported, shown as follows:
Example:
openssl s_client -cipher ?<some cipher to test>? -connect <apic ipaddress>:<ssl port, usually 443>
See the following example cipher:
Example:
openssl s_client -cipher ?ECDHE-ECDSA-AES128-GCM-SHA256? -connect 10.1.1.14:443
Note
If the response contains CONNECTED, then the cipher is supported.
Cisco APIC Troubleshooting Guide
120
CHAPTER
17
Cisco APIC Troubleshooting Guide
121
Removing Unwanted _ui_ Objects
Removing Unwanted _ui_ Objects
Caution
Changes made through the APIC Basic GUI can be seen, but cannot be modified in the Advanced GUI,
and changes made in the Advanced GUI cannot be rendered in the Basic GUI. The Basic GUI is kept
synchronized with the NX-OS style CLI, so that if you make a change from the NX-OS style CLI, these
changes are rendered in the Basic GUI, and changes made in the Basic GUI are rendered in the NX-OS
style CLI, but the same synchronization does not occur between the Advanced GUI and the NX-OS style
CLI. See the following examples:
• Do not mix Basic and Advanced GUI modes. If you apply an interface policy to two ports using
Advanced mode and then change the settings of one port using Basic mode, your changes might be
applied to both ports.
• Do not mix the Advanced GUI and the CLI, when doing per-interface configuration on APIC.
Configurations performed in the GUI, may only partially work in the NX-OS CLI.
For example, if you configure a switch port in the GUI at Tenants > tenant-name > Application
Profiles > application-profile-name > Application EPGs > EPG-name > Static Ports > Deploy
Static EPG on PC, VPC, or Interface
Then you use the show running-config command in the NX-OS style CLI, you receive output such
as:
leaf 102
interface ethernet 1/15
switchport trunk allowed vlan 201 tenant t1 application ap1 epg ep1
exit
exit
If you use these commands to configure a static port in the NX-OS style CLI, the following error
occurs:
apic1(config)# leaf 102
apic1(config-leaf)# interface ethernet 1/15
apic1(config-leaf-if)# switchport trunk allowed vlan 201 tenant t1 application ap1
epg ep1
No vlan-domain associated to node 102 interface ethernet1/15 encap vlan-201
This occurs because the CLI has validations that are not performed by the APIC GUI. For the
commands from the show running-config command to function in the NX-OS CLI, a vlan-domain
must have been previously configured. The order of configuration is not enforced in the GUI.
• Do not make changes with the Basic GUI or the NX-OS CLI before using the Advanced GUI. This
may also inadvertantly cause objects to be created (with names prepended with _ui_) which cannot
be changed or deleted in the Advanced GUI.
If you make changes with the Basic GUI or the NX-OS CLI before using the Advanced GUI, this may
inadvertently cause objects to be created (with names prepended with _ui_) which cannot be changed or
deleted in the Advanced GUI.
For the steps to remove such objects, see Removing Unwanted _ui_ Objects Using the REST API, on page
123.
• Removing Unwanted _ui_ Objects Using the REST API, page 123
Cisco APIC Troubleshooting Guide
122
Removing Unwanted _ui_ Objects
Removing Unwanted _ui_ Objects Using the REST API
Removing Unwanted _ui_ Objects Using the REST API
If you make changes with the Basic GUI or the NX-OS CLI before using the Advanced GUI, and objects
appear in the Advanced GUI (with names prepended with _ui_), these objects can be removed by performing
a REST API request to the API, containing the following:
• The Class name, for example infraAccPortGrp
• The Dn attribute, for example dn="uni/infra/funcprof/accportgrp-__ui_l101_eth1--31"
• The Status attribute set to status="deleted"
Perform the POST to the API with the following steps:
Step 1
Step 2
Log on to a user account with write access to the object to be removed.
Send a POST to the API such as the following example:
POST https://192.168.20.123/api/mo/uni.xml
Payload:<infraAccPortGrp dn="uni/infra/funcprof/accportgrp-__ui_l101_eth1--31" status="deleted"/>
Cisco APIC Troubleshooting Guide
123
Removing Unwanted _ui_ Objects
Removing Unwanted _ui_ Objects Using the REST API
Cisco APIC Troubleshooting Guide
124
APPENDIX
A
acidiag Command
To troubleshoot operations on the Cisco APIC, use the acidiag command.
Caution
This command is not intended for every day operation of ACI. Running all forms of the command can be
very disruptive and cause major issues in your network if not used properly. Make sure you understand
the full effect on your fabric before running them.
Cluster Commands
acidiag
acidiag avread
acidiag fnvread
acidiag fnvreadex
Cisco APIC Troubleshooting Guide
125
acidiag Command
Syntax Description
Option
Function
avread
Displays APICs within the cluster. The avread output
includes:
• Cluster of —Operational cluster size
• out of targeted—The desired cluster size
• active= —Indicates whether the APIC is
reachable
• health= —The overall APIC health summary.
Displays services with degraded health scores.
• chassisID= —The known chassis IDs for a
given APIC.
Note
bootcurr
On the next boot, the APIC system will boot the
current APIC image in the Linux partition. This
option is not expected to normally be used.
bootother
On the next boot, the APIC system will boot the
previous APIC image in the Linux partition. This
option is not expected to normally be used.
bond0test
Disruptive test of the APIC connection to the leaf.
This is used for internal Cisco testing purposes only
and outside of that could cause issues with the APIC
connection to the fabric.
fnvread
Displays the address and state of switch nodes
registered with the fabric.
fnvreadex
Displays additional information for switch nodes
registered with the fabric.
linkflap
Brings down and back up a specified APIC interface.
preservelogs
APIC will archive current logs. During a normal
reboot this automatically occurs. This option can be
used prior to a hard reboot.
run
Two available options are iptables-list and lldptool.
The iptables-list is used to display the Linux iptables,
which are controlled by the mgmt Tenant contracts.
lldptool is used to display lldp information which is
sent or received by the APIC.
rvread
Summarizes the data layer state. The output shows
a summary of the data layer state for each service.
The shard view shows replicas in ascending order.
Cisco APIC Troubleshooting Guide
126
Peer chassis IDs can be incorrect for
APICs not currently in the cluster.
acidiag Command
Option
Function
acidiag rvread service
Displays the data layer state for a service on all shards
across all replicas.
Note
For an example, see Examples, on page
130
acidiag rvread service shard
Displays the data layer state for a service on a specific
shard across all replicas.
Note
For an example, see Examples, on page
130
acidiag rvread service shard replica
Displays the data layer state for a service on a specific
shard and replica.
Note
For an example, see Examples, on page
130
validateimage
Prior to loading an image into the firmware
repository, the image can be validated. Note that this
function runs as a normal part of the process of the
image being added into the repository.
validateenginxconf
Validates the generated nginx configuration file on
APIC to ensure nginx can start with that configuration
file. This is meant for debug use, in cases where the
nginx webserver is not running on APIC.
Service IDs
The service IDs listed in the table below are also visible when entering the man acidiag command.
Table 2: Service IDs
Service
ID
cliD
1
controller
2
eventmgr
3
extXMLApi
4
policyelem
5
policymgr
6
reader
7
ae
8
topomgr
9
Cisco APIC Troubleshooting Guide
127
acidiag Command
Service
ID
observer
10
dbgr
11
observerelem
12
dbgrelem
13
vmmmgr
14
nxosmock
15
bootmgr
16
appliancedirector
17
adrelay
18
ospaagent
19
vleafelem
20
dhcpd
21
scripthandler
22
idmgr
23
ospaelem
24
osh
25
opflexagent
26
opflexelem
27
confelem
28
vtap
29
snmpd
30
opflexp
31
analytics
32
policydist
33
Cisco APIC Troubleshooting Guide
128
acidiag Command
Service
ID
plghandler
34
Table 3: Data States
State
ID
COMATOSE
0
NEWLY_BORN
1
UNKNOWN
2
DATA_LAYER_DIVERGED
11
DATA_LAYER_DEGRADED_LEADERSHIP
12
DATA_LAYER_ENTIRELY_DIVERGED
111
DATA_LAYER_PARTIALLY_DIVERGED
112
DATA_LAYER_ENTIRELY_DEGRADED_LEADERSHIP
121
DATA_LAYER_PARTIALLY_DEGRADED_LEADERSHIP
122
FULLY_FIT
255
System Keywords
acidiag [start| stop| restart] [mgmt| xinetd]
acidiag installer -u imageurl -c
acidiag reboot
acidiag touch [clean| setup]
acidiag verifyapic
Syntax Description
Option
Function
-c
Specifies a clean install
-u
Specifies a URL for the APIC image.
imageurl
Specifies an APIC image.
installer
Installs a new image on the APIC, -c for clean install
mgmt
Specifies all services on the APIC.
Cisco APIC Troubleshooting Guide
129
acidiag Command
Option
Function
reboot
Reboots the APIC.
restart
Restarts services on an APIC.
start
Starts services on an APIC.
stop
Stops services on an APIC.
touch [clean | setup]
Resets the APIC configuration.
• The clean option removes all policy data while
retaining the APIC network configuration (such
as fabric name, IP address, login)
• The setup option removes both policy data and
the APIC network configuration.
verifyapic
Displays the APIC software version.
xinetd
Specifies xinetd (extended internet daemon) service,
which controls the ssh and telnet daemons.
Diagnostic Keywords
acidiag crashsuspecttracker
acidiag dbgtoken
acidiag version
Syntax Description
Option
Function
crashsuspecttracker
Tracks states of a service or data subset that indicate
a crash.
dbgtoken
Generates a token used to generate a root password.
This is to be used as directed while working with the
TAC as needed.
version
Displays the APIC ISO software version.
Examples
The following examples show how to use the acidiag command:
apic1# acidiag version 2.2.1o
apic1# acidiag verifyapic
openssl_check: certificate details
subject= CN=ABC12345678,serialNumber=PID:APIC-SERVER-L1 SN:ABC12345678
issuer= CN=Cisco Manufacturing CA,O=Cisco Systems
notBefore=Sep 28 17:17:42 2016 GMT
Cisco APIC Troubleshooting Guide
130
acidiag Command
notAfter=Sep 28 17:27:42 2026 GMT
openssl_check: passed
ssh_check: passed
all_checks: passed
apic1# acidiag avread
Local appliance ID=1 ADDRESS=10.0.0.1 TEP ADDRESS=10.0.0.0/16
CHASSIS_ID=10220833-ea00-3bb3-93b2-ef1e7e645889
Cluster of 3 lm(t):1(2014-07-12T19:54:04.877+00:00) appliances
(out of targeted 3 lm(t):3(2014-07-12T19:55:03.442+00:00))
with FABRIC_DOMAIN name=mininet set to version=1.0(0.414)
lm(t):3(2014-07-12T19:55:13.564+00:00)
appliance id=1 last mutated at 2014-07-12T19:46:06.831+00:00 address=10.0.0.1 tep
address=10.0.0.0/16
oob address=192.168.10.1/24 version=1.0(0.414) lm(t):1(2014-07-12T19:54:05.146+00:00)
chassisId=10220833-ea00-3bb3-93b2-ef1e7e645889 lm(t):1(2014-07-12T19:54:05.146+00:00)
commissioned=1 registered=1 active=yes(zeroTime)
health=(applnc:255 lm(t):1(2014-07-12T20:01:22.934+00:00) svc's)
appliance id=2 last mutated at 2014-07-12T19:51:10.649+00:00 address=10.0.0.2 tep
address=10.0.0.0/16
oob address=192.168.10.2/24 version=1.0(0.414) lm(t):2(2014-07-12T19:54:05.064+00:00)
chassisId=5d74122c-2ab9-3ccb-b06d-f620d5e20ccd lm(t):2(2014-07-12T19:54:05.064+00:00)
commissioned=1 registered=1 active=yes(2014-07-12T19:51:10.651+00:00)
health=(applnc:255 lm(t):2(2014-07-12T20:01:22.442+00:00) svc's)
appliance id=3 last mutated at 2014-07-12T19:54:05.028+00:00 address=10.0.0.3 tep
address=10.0.0.0/16
oob address=192.168.10.3/24 version=1.0(0.414) lm(t):3(2014-07-12T19:54:05.361+00:00)
chassisId=71355d49-6fe7-3a78-a361-72d6c1e3360c lm(t):3(2014-07-12T19:54:05.361+00:00)
commissioned=1 registered=1 active=yes(2014-07-12T19:54:05.029+00:00)
health=(applnc:255 lm(t):3(2014-07-12T20:01:22.892+00:00) svc's)
clusterTime=<diff=0 common=2014-07-14T16:52:20.343+00:00 local=2014-07-14T16:52:20.343+00:00
pF=<displForm=0
offsSt=0 offsVlu=0 lm(t):3(2014-07-12T19:55:03.750+00:00)>>
--------------------------------------------apic1# acidiag rvread 6 3 1
(6,3,1) st:6 lm(t):3(2014-10-16T08:48:20.238+00:00) le: reSt:LEADER voGr:0 cuTerm:0x19
lCoTe:0x18
lCoIn:0x1800000000001b2a veFiSt:0x31 veFiEn:0x31 lm(t):3(2014-10-16T08:48:20.120+00:00)
lastUpdt 2014-10-16T09:07:00.214+00:00
--------------------------------------------clusterTime=<diff=65247252 common=2014-10-16T09:07:01.837+00:00
local=2014-10-15T14:59:34.585+00:00
pF=<displForm=0 offsSt=0 offsVlu=0 lm(t):3(2014-10-16T04:50:08.714+00:00)>>
apic1# acidiag rvread 6 3
(6,3,1) st:6 lm(t):3(2014-10-16T08:48:20.238+00:00) le: reSt:LEADER voGr:0 cuTerm:0x19
lCoTe:0x18
lCoIn:0x1800000000001b2a veFiSt:0x31 veFiEn:0x31 lm(t):3(2014-10-16T08:48:20.120+00:00)
lastUpdt 2014-10-16T09:08:30.240+00:00
(6,3,2) st:6 lm(t):1(2014-10-16T08:47:25.323+00:00) le: reSt:FOLLOWER voGr:0 cuTerm:0x19
lCoTe:0x18
lCoIn:0x1800000000001b2a veFiSt:0x49 veFiEn:0x49 lm(t):1(2014-10-16T08:48:20.384+00:00)
lp: clSt:2
lm(t):1(2014-10-16T08:47:03.286+00:00) dbSt:2 lm(t):1(2014-10-16T08:47:02.143+00:00)
stMmt:1
lm(t):0(zeroTime) dbCrTs:2014-10-16T08:47:02.143+00:00 lastUpdt
2014-10-16T08:48:20.384+00:00
(6,3,3) st:6 lm(t):2(2014-10-16T08:47:13.576+00:00) le: reSt:FOLLOWER voGr:0 cuTerm:0x19
lCoTe:0x18
lCoIn:0x1800000000001b2a veFiSt:0x43 veFiEn:0x43 lm(t):2(2014-10-16T08:48:20.376+00:00)
lastUpdt 2014-10-16T09:08:30.240+00:00
--------------------------------------------clusterTime=<diff=65247251 common=2014-10-16T09:08:30.445+00:00
Cisco APIC Troubleshooting Guide
131
acidiag Command
local=2014-10-15T15:01:03.194+00:00
pF=<displForm=0 offsSt=0 offsVlu=0 lm(t):3(2014-10-16T04:50:08.714+00:00)>>
Cisco APIC Troubleshooting Guide
132
APPENDIX
B
Configuring Export Policies for Troubleshooting
Export policies enable you to export statistics, technical support collections, faults, and events to process
core files and debug data from the fabric (the APIC as well as the switch) to any external host.
• About Exporting Files, page 133
• File Export Guidelines and Restrictions, page 133
• Configuring a Remote Location, page 134
• Sending an On-Demand Tech Support File, page 136
About Exporting Files
An administrator can configure export policies in the APIC to export statistics, technical support collections,
faults and events, to process core files and debug data from the fabric (the APIC as well as the switch) to any
external host. The exports can be in a variety of formats, including XML, JSON, web sockets, secure copy
protocol (SCP), or HTTP. You can subscribe to exports in streaming, periodic, or on-demand formats.
An administrator can configure policy details such as the transfer protocol, compression algorithm, and
frequency of transfer. Policies can be configured by users who are authenticated using AAA. A security
mechanism for the actual transfer is based on a username and password. Internally, a policy element handles
the triggering of data.
File Export Guidelines and Restrictions
• HTTP export and the streaming API format is supported only with statistics information. Core and Tech
Support data are not supported.
• The destination IP for exported files cannot be an IPv6 address.
Cisco APIC Troubleshooting Guide
133
Configuring Export Policies for Troubleshooting
Configuring a Remote Location
Note
Do not trigger Tech Support from more than five nodes simultaneously, especially if they are to be
exported into the APIC or to an external server with insufficient bandwidth and compute resources.
In order to collect Tech Support from all the nodes in the fabric periodically, you must create multiple
policies. Each policy must cover a subset of the nodes and should be scheduled to trigger in a staggered
way (at least 30 minutes apart).
Configuring a Remote Location
Configuring a Remote Location Using the GUI
This procedure explains how to create a remote location using the APIC GUI.
Step 1
Step 2
On the menu bar, choose ADMIN > Import/Export.
In the navigation pane, right-click Remote Locations and choose Create Remote Location.
The Create Remote Location dialog appears.
Step 3
Enter the appropriate values in the Create Remote Location dialog fields.
Note
For an explanation of a field, click the 'i' icon to display the help
file.
When finished entering values in the Create Remote Location dialog fields, click Submit.
You have now created a remote location for backing up your data.
Step 4
Configuring a Remote Location Using the REST API
This procedure explains how to create a remote location using the REST API.
<fileRemotePath name="local" host=“host or ip" protocol=“ftp|scp|sftp" remotePath=“path to
folder" userName=“uname" userPasswd=“pwd" />
Configuring a Remote Location Using the NX-OS Style CLI
In the ACI fabric, you can configure one or more remote destinations for exporting techsupport or configuration
files.
SUMMARY STEPS
1. configure
2. [no] remote path remote-path-name
3. user username
4. path {ftp | scp | sftp} host[ :port ] [remote-directory ]
Cisco APIC Troubleshooting Guide
134
Configuring Export Policies for Troubleshooting
Configuring a Remote Location Using the NX-OS Style CLI
DETAILED STEPS
Step 1
Command or Action
Purpose
configure
Enters global configuration mode.
Example:
apic1# configure
Step 2
[no] remote path remote-path-name
Enters configuration mode for a remote path.
Example:
apic1(config)# remote path myFiles
Step 3
user username
Sets the user name for logging in to the remote server.
You are prompted for a password.
Example:
apic1(config-remote)# user admin5
Step 4
path {ftp | scp | sftp} host[ :port ] [remote-directory ]
Sets the path and protocol to the remote server. You
are prompted for a password.
Example:
apic1(config-remote)# path sftp
filehost.example.com:21 remote-directory
/reports/apic
Examples
This example shows how to configure a remote path for exporting files.
apic1# configure
apic1(config)# remote path myFiles
apic1(config-remote)# user admin5
You must reset the password when modifying the path:
Password:
Retype password:
apic1(config-remote)# path sftp filehost.example.com:21 remote-directory /reports/apic
You must reset the password when modifying the path:
Password:
Retype password:
Cisco APIC Troubleshooting Guide
135
Configuring Export Policies for Troubleshooting
Sending an On-Demand Tech Support File
Sending an On-Demand Tech Support File
Sending an On-Demand Techsupport File Using the GUI
Step 1
Step 2
Step 3
In the menu bar, click Admin.
In the submenu bar, click Import/Export.
In the Navigation pane, expand Export Policies.
Step 4
Right-click On-demand TechSupport and choose Create On-demand TechSupport.
The Create On-demand TechSupport dialog box appears.
Step 5
Enter the appropriate values in the fields of the Create On-demand TechSupport dialog box.
Note
For an explanation of a field, click the information (i) icon in the Create On-demand TechSupport dialog
box. The help file opens to a properties description page.
Click Submit to send the techsupport file.
Note
On-demand tech support files can be saved to another APIC to balance storage and CPU requirements. To verify
the location, click on the On-demand TechSupport policy in the Navigation pane, then click the OPERATIONAL
tab in the Work pane. The controller is displayed in the EXPORT LOCATION field.
Right-click the policy name and choose Collect Tech Support.
Choose Yes to begin collecting tech support information.
Step 6
Step 7
Step 8
Sending an On-Demand TechSupport File Using the REST API
Step 1
Set the remote destination for a technical support file using the REST API, by sending a POST with XML such as the
following example:
Example:
<fileRemotePath userName="" remotePort="22" remotePath="" protocol="sftp" name="ToSupport"
host="192.168.200.2"
dn="uni/fabric/path-ToSupport" descr="">
<fileRsARemoteHostToEpg tDn="uni/tn-mgmt/mgmtp-default/oob-default"/>
</fileRemotePath>
Step 2
Generate an on-demand technical support file using the REST API by sending a POST with XML such as the following:
Example:
<dbgexpTechSupOnD upgradeLogs="no" startTime="unspecified" name="Tech_Support_9-20-16"
exportToController="no"
endTime="unspecified" dn="uni/fabric/tsod-Tech_Support_9-20-16" descr="" compression="gzip"
category="forwarding" adminSt="untriggered">
<dbgexpRsExportDest tDn="uni/fabric/path-ToSupport"/>
Cisco APIC Troubleshooting Guide
136
Configuring Export Policies for Troubleshooting
Sending an On-Demand TechSupport File Using the REST API
<dbgexpRsTsSrc tDn="topology/pod-1/node-102/sys"/>
<dbgexpRsTsSrc tDn="topology/pod-1/node-103/sys"/>
<dbgexpRsTsSrc tDn="topology/pod-1/node-101/sys"/>
<dbgexpRsData tDn="uni/fabric/tscont"/>
</dbgexpTechSupOnD>
Cisco APIC Troubleshooting Guide
137
Configuring Export Policies for Troubleshooting
Sending an On-Demand TechSupport File Using the REST API
Cisco APIC Troubleshooting Guide
138
APPENDIX
C
Finding the Switch Inventory
Knowing your switch model and serial numbers can help TAC support with troubleshooting your fabric.
This section explains how to find the switch model and serial numbers using the Cisco APIC GUI, CLI, and
REST API.
• Finding Your Switch Inventory Using the GUI, page 139
• Finding Your Switch Inventory Using the NX-OS CLI, page 140
• Finding Your Switch Inventory Using the REST API, page 142
Finding Your Switch Inventory Using the GUI
This section explains how to find your switch model and serial numbers using the Cisco APIC GUI.
Before You Begin
You must have access to the Cisco APIC GUI
Step 1
Step 2
Step 3
Step 4
On the menu bar, choose Fabric > Inventory.
In the navigation pane, click a Pod icon.
Your switch icons appear in the navigation pane.
In the navigation pane, click on a switch icon.
A list of tabs appears at the top of the work pane.
Click the General tab.
Your switch information appears in the work pane.
Cisco APIC Troubleshooting Guide
139
Finding the Switch Inventory
Finding Your Switch Inventory Using the NX-OS CLI
Finding Your Switch Inventory Using the NX-OS CLI
This section explains how to find your switch model and serial numbers using the NX-OS CLI.
Find your switch inventory as follows:
Example:
switch# show hardware
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
Software
BIOS:
version 07.56
kickstart: version 12.1(1h) [build 12.1(1h)]
system:
version 12.1(1h) [build 12.1(1h)]
PE:
version 2.1(1h)
BIOS compile time:
06/08/2016
kickstart image file is: /bootflash/aci-n9000-dk9.12.1.1h.bin
kickstart compile time: 10/01/2016 20:10:40 [10/01/2016 20:10:40]
system image file is:
/bootflash/auto-s
system compile time:
10/01/2016 20:10:40 [10/01/2016 20:10:40]
Hardware
cisco N9K-C93180YC-EX ("supervisor")
Intel(R) Xeon(R) CPU @ 1.80GHz with 16400384 kB of memory.
Processor Board ID FDO20101H1W
Device name: ifav41-leaf204
bootflash:
62522368 kB
Kernel uptime is 02 day(s), 21 hour(s), 42 minute(s), 31 second(s)
Last reset at 241000 usecs after Sun Oct 02 01:27:25 2016
Reason: reset-by-installer
System version: 12.1(1e)
Service: Upgrade
plugin
Core Plugin, Ethernet Plugin
-------------------------------Switch hardware ID information
-------------------------------Switch is booted up
Switch type is : Nexus C93180YC-EX Chassis
Model number is N9K-C93180YC-EX
H/W version is 0.2010
Part Number is 73-15298-01
Part Revision is 1
Manufacture Date is Year 20 Week 10
Serial number is FDO20101H1W
CLEI code is 73-15298-01
--------------------------------
Cisco APIC Troubleshooting Guide
140
Finding the Switch Inventory
Finding Your Switch Inventory Using the NX-OS CLI
Chassis has one slot
-------------------------------Module1 ok
Module type is : 48x10/25G
1 submodules are present
Model number is N9K-C93180YC-EX
H/W version is 0.2110
Part Number is 73-17776-02
Part Revision is 11
Manufacture Date is Year 20 Week 10
Serial number is FDO20101H1W
CLEI code is 73-17776-02
GEM ok
Module type is : 6x40/100G Switch
1 submodules are present
Model number is N9K-C93180YC-EX
H/W version is 0.2110
Part Number is 73-17776-02
Part Revision is 11
Manufacture Date is Year 20 Week 10
Serial number is FDO20101H1W
CLEI code is 73-17776-02
--------------------------------------Chassis has 2 PowerSupply Slots
--------------------------------------PS1 shut
Power supply type is : 54.000000W 220v AC
Model number is NXA-PAC-650W-PE
H/W version is 0.0
Part Number is 341-0729-01
Part Revision is A0
Manufacture Date is Year 19 Week 50
Serial number is LIT19500ZEK
CLEI code is 341-0729-01
PS2 ok
Power supply type is : 54.000000W 220v AC
Model number is NXA-PAC-650W-PE
H/W version is 0.0
Part Number is 341-0729-01
Part Revision is A0
Manufacture Date is Year 19 Week 50
Serial number is LIT19500ZEA
CLEI code is 341-0729-01
--------------------------------------Chassis has 4 Fans
--------------------------------------FT1 ok
Fan1(sys_fan1)(fan_model:NXA-FAN-30CFM-F)
is not available
is inserted but info
FT2 ok
Fan2(sys_fan2)(fan_model:NXA-FAN-30CFM-F)
is not available
is inserted but info
FT3 ok
Fan3(sys_fan3)(fan_model:NXA-FAN-30CFM-F)
is not available
is inserted but info
FT4 ok
Fan4(sys_fan4)(fan_model:NXA-FAN-30CFM-F)
is inserted but info
Cisco APIC Troubleshooting Guide
141
Finding the Switch Inventory
Finding Your Switch Inventory Using the REST API
is not available
====================================================================================
Finding Your Switch Inventory Using the REST API
This section explains how to find your switch model and serial numbers using the REST API
Find your switch inventory as follows:
Example:
GET
https://192.0.20.123/api/node/mo/topology/pod-1.json?query-target=children&target-subtree-class=fabricNode
The following response is returned:
response:
{
"totalCount":"8",
"imdata":
[{
"fabricNode":{
"attributes":{
"adSt":"on",
"childAction":"",
"delayedHeartbeat":"no",
"dn":"topology/pod-1/node-103",
"fabricSt":"active",
"id":"103",
"lcOwn":"local",
"modTs":"2016-10-08T14:49:35.665+00:00",
"model":"N9K-C9396PX",
"monPolDn":"uni/fabric/monfab-default",
"name":"leaf3",
"nameAlias":"",
"role":"leaf",
"serial":"TEP-1-103",
"status":"","uid":"0",
"vendor":"Cisco Systems, Inc",
"version":""}
}
},{
"fabricNode":{
"attributes":{
"adSt":"on",
"childAction":"",
"delayedHeartbeat":"no",
"dn":"topology/pod-1/node-105",
"fabricSt":"active",
Cisco APIC Troubleshooting Guide
142
Finding the Switch Inventory
Finding Your Switch Inventory Using the REST API
"id":"105",
"lcOwn":"local",
"modTs":"2016-10-08T14:47:52.011+00:00",
"model":"N9K-C9508",
"monPolDn":"uni/fabric/monfab-default",
"name":"spine2",
"nameAlias":"",
"role":"spine",
"serial":"TEP-1-105","status":"",
"uid":"0",
"vendor":"Cisco Systems, Inc",
"version":""
...
[TRUNCATED]
...
}
Cisco APIC Troubleshooting Guide
143
Finding the Switch Inventory
Finding Your Switch Inventory Using the REST API
Cisco APIC Troubleshooting Guide
144
APPENDIX
D
Cisco APIC Cluster Management
This section explains how to expand, contract, commission, and decommission Cisco APIC clusters. For
more information about Cisco APIC clusters, see the Cisco APIC Cluster Management document:
http://www.cisco.com/c/en/us/support/cloud-systems-management/
application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html
This section contains the following topics:
• Expanding the Cisco APIC Cluster, page 145
• Contracting the Cisco APIC Cluster , page 146
• Cluster Management Guidelines, page 146
• Expanding the Cluster Examples, page 149
• Contracting the Cluster Examples, page 150
• Commissioning and Decommissioning Cisco APIC Controllers, page 152
• Replacing a Cisco APIC in a Cluster Using the CLI, page 153
Expanding the Cisco APIC Cluster
Expanding the Cisco APIC cluster is the operation to increase any size mismatches, from a cluster size of N
to size N+1, within legal boundaries. The operator sets the administrative cluster size and connects the APICs
with the appropriate cluster IDs, and the cluster performs the expansion.
During cluster expansion, regardless of in which order you physically connect the APIC controllers, the
discovery and expansion takes place sequentially based on the APIC ID numbers. For example, APIC2 is
discovered after APIC1, and APIC3 is discovered after APIC2 and so on until you add all the desired APICs
to the cluster. As each sequential APIC is discovered, a single data path or multiple data paths are established,
and all the switches along the path join the fabric. The expansion process continues until the operational cluster
size reaches the equivalent of the administrative cluster size.
Cisco APIC Troubleshooting Guide
145
Cisco APIC Cluster Management
Contracting the Cisco APIC Cluster
Contracting the Cisco APIC Cluster
Contracting the Cisco APIC cluster is the operation to decrease any size mismatches, from a cluster size of
N to size N -1, within legal boundaries. As the contraction results in increased computational and memory
load for the remaining APICs in the cluster, the decommissioned APIC cluster slot becomes unavailable by
operator input only.
During cluster contraction, you must begin decommissioning the last APIC in the cluster first and work your
way sequentially in reverse order. For example, APIC4 must be decommissioned before APIC3, and APIC3
must be decommissioned before APIC2.
Cluster Management Guidelines
The APIC cluster is comprised of multiple APIC controllers that provide operators a unified real time
monitoring, diagnostic, and configuration management capability for the ACI fabric. To assure optimal system
performance, follow the guidelines below for making changes to the APIC cluster.
Note
Prior to initiating a change to the cluster, always verify its health. When performing planned changes to
the cluster, all controllers in the cluster should be healthy. If one or more of the APIC controllers' health
status in the cluster is not "fully fit", remedy that situation before proceeding. Also, assure that cluster
controllers added to the APIC are running the same version of firmware as the other controllers in the
APIC cluster. See the Cisco APIC Troubleshooting Guide for more information on resolving APIC cluster
health issues.
Follow these general guidelines when managing clusters:
• It is recommended that you have at least 3 active APICs in a cluster, and one or more standby APICs.
• Disregard cluster information from APICs that are not currently in the cluster; they do not provide
accurate cluster information.
• Cluster slots contain an APIC ChassisID. Once you configure a slot, it remains unavailable until you
decommission the APIC with the assigned ChassisID.
• If an APIC firmware upgrade is in progress, wait for it to complete and the cluster to be fully fit before
proceeding with any other changes to the cluster.
• Always decommission an APIC cluster controller before doing a power cycle, then add it back to the
cluster. Failure to follow this guideline can corrupt the APIC cluster database shards residing on the
cluster, which will require doing a wipe of the controller, then a cluster synchronization to restore a valid
copy of the APIC cluster database from the other APIC controllers in the cluster.
• When an APIC cluster is split into two or more groups, the ID of a node is changed and the changes are
not synchronized across all APICs. This can cause inconsistency in the node IDs between APICs and
also the affected leaf nodes may not appear in the inventory in the APIC GUI. When you split an APIC
cluster, decommission the affected leaf nodes from APIC and register them again, so that the inconsistency
in the node IDs is resolved and the health status of the APICs in a cluster are in a fully fit state
• Before configuring the APIC cluster, ensure that all the APICs are running the same firmware version.
Initial clustering of APICs running differing versions is an unsupported operation and may cause problems
within the cluster.
Cisco APIC Troubleshooting Guide
146
Cisco APIC Cluster Management
Expanding the APIC Cluster Size
This section contains the following topics:
Expanding the APIC Cluster Size
Follow these guidelines to expand the APIC cluster size:
• Schedule the cluster expansion at a time when the demands of the fabric workload will not be impacted
by the cluster expansion.
• If one or more of the APIC controllers' health status in the cluster is not "fully fit", remedy that situation
before proceeding.
• Stage the new APIC controller(s) according to the instructions in their hardware installation guide. Verify
in-band connectivity with a PING test.
• Increase the cluster target size to be equal to the existing cluster size controller count plus the new
controller count. For example, if the existing cluster size controller count is 3 and you are adding 3
controllers, set the new cluster target size to 6. The cluster proceeds to sequentially increase its size one
controller at a time until all new the controllers are included in the cluster.
Note
Cluster expansion stops if an existing APIC controller becomes unavailable. Resolve
this issue before attempting to proceed with the cluster expansion.
• Depending on the amount of data the APIC must synchronize upon the addition of each appliance, the
time required to complete the expansion could be more than 10 minutes per appliance. Upon successful
expansion of the cluster, the APIC operational size and the target size will be equal.
Note
Allow the APIC to complete the cluster expansion before making additional changes
to the cluster.
Reducing the APIC Cluster Size
Follow these guidelines to reduce the APIC cluster size and decommission the APIC controllers that are
removed from the cluster:
Note
Failure to follow an orderly process to decommission and power down APIC controllers from a reduced
cluster can lead to unpredictable outcomes. Do not allow unrecognized APIC controllers to remain
connected to the fabric.
• Reducing the cluster size increases the load on the remaining APIC controllers. Schedule the APIC
controller size reduction at a time when the demands of the fabric workload will not be impacted by the
cluster synchronization.
• If one or more of the APIC controllers' health status in the cluster is not "fully fit", remedy that situation
before proceeding.
Cisco APIC Troubleshooting Guide
147
Cisco APIC Cluster Management
Replacing Cisco APIC Controllers in the Cluster
• Reduce the cluster target size to the new lower value. For example if the existing cluster size is 6 and
you will remove 3 controllers, reduce the cluster target size to 3.
• Starting with the highest numbered controller ID in the existing cluster, decommission, power down,
and disconnect the APIC controller one by one until the cluster reaches the new lower target size.
Upon the decommissioning and removal of each controller, the APIC synchronizes the cluster.
Note
After decommissioning an APIC controller from the cluster, power it down and
disconnect it from fabric. Before returning it to service, do a wiped clean back to factory
reset.
• Cluster synchronization stops if an existing APIC controller becomes unavailable. Resolve this issue
before attempting to proceed with the cluster synchronization.
• Depending on the amount of data the APIC must synchronize upon the removal of a controller, the time
required to decommission and complete cluster synchronization for each controller could be more than
10 minutes per controller.
Note
Complete the entire necessary decommissioning steps, allowing the APIC to complete the cluster
synchronization accordingly before making additional changes to the cluster.
Replacing Cisco APIC Controllers in the Cluster
Follow these guidelines to replace Cisco APIC controllers:
• If the health status of any Cisco APIC controller in the cluster is not Fully Fit, remedy the situation
before proceeding.
• Schedule the Cisco APIC controller replacement at a time when the demands of the fabric workload
will not be impacted by the cluster synchronization.
• Make note of the initial provisioning parameters and image used on the Cisco APIC controller that will
be replaced. The same parameters and image must be used with the replacement controller. The Cisco
APIC proceeds to synchronize the replacement controller with the cluster.
Note
Cluster synchronization stops if an existing Cisco APIC controller becomes unavailable.
Resolve this issue before attempting to proceed with the cluster synchronization.
• You must choose a Cisco APIC controller that is within the cluster and not the controller that is being
decommissioned. For example: Log in to Cisco APIC1 or APIC2 to invoke the shutdown of APIC3 and
decommission APIC3.
• Perform the replacement procedure in the following order:
1 Make note of the configuration parameters and image of the APIC being replaced.
2 Decommission the APIC you want to replace (see Decommissioning a Cisco APIC Controller in the
Cluster Using the GUI, on page 152)
Cisco APIC Troubleshooting Guide
148
Cisco APIC Cluster Management
Expanding the Cluster Examples
3 Commission the replacement APIC using the same configuration and image of the APIC being
replaced (see Commissioning a Cisco APIC Controller in the Cluster Using the GUI, on page 152)
• Stage the replacement Cisco APIC controller according to the instructions in its hardware installation
guide. Verify in-band connectivity with a PING test.
Note
Failure to decommission Cisco APIC controllers before attempting their replacement
will preclude the cluster from absorbing the replacement controllers. Also, before
returning a decommissioned Cisco APIC controller to service, do a wiped clean back
to factory reset.
• Depending on the amount of data the Cisco APIC must synchronize upon the replacement of a controller,
the time required to complete the replacement could be more than 10 minutes per replacement controller.
Upon successful synchronization of the replacement controller with the cluster, the Cisco APIC operational
size and the target size will remain unchanged.
Note
Allow the Cisco APIC to complete the cluster synchronization before making additional
changes to the cluster.
• The UUID and fabric domain name persist in a Cisco APIC controller across reboots. However, a clean
back-to-factory reboot removes this information. If a Cisco APIC controller is to be moved from one
fabric to another, a clean back-to-factory reboot must be done before attempting to add such an controller
to a different Cisco ACI fabric.
Expanding the Cluster Examples
Expanding the APIC Cluster Using the GUI
Step 1
Step 2
Step 3
Step 4
Step 5
On the menu bar, choose SYSTEM > Controllers. In the Navigation pane, expand Controllers > apic_controller_name
> Cluster.
You must choose an apic_controller_name that is within the cluster that you wish to expand.
In the Work pane, the cluster details are displayed. This includes the current cluster target and current sizes, the
administrative, operational, and health states of each controller in the cluster.
Verify that the health state of the cluster is Fully Fit before you proceed with contracting the cluster.
In the Work pane, click Actions > Change Cluster Size.
In the Change Cluster Size dialog box, in the Target Cluster Administrative Size field, choose the target number to
which you want to expand the cluster. Click Submit.
Note
It is not acceptable to have a cluster size of two APIC controllers. A cluster of one, three, or more APIC controllers
is acceptable.
In the Confirmation dialog box, click Yes.
Cisco APIC Troubleshooting Guide
149
Cisco APIC Cluster Management
Expanding the APIC Cluster Using the REST API
In the Work pane, under Properties, the Target Size field must display your target cluster size.
Step 6
Step 7
Physically connect all the APIC controllers that are being added to the cluster.
In the Work pane, in the Cluster > Controllers area, the APIC controllers are added one by one and displayed in the
sequential order starting with N + 1 and continuing until the target cluster size is achieved.
Verify that the APIC controllers are in operational state, and the health state of each controller is Fully Fit.
Expanding the APIC Cluster Using the REST API
The cluster drives its actual size to the target size. If the target size is higher than the actual size, the cluster
size expands.
Step 1
Set the target cluster size to expand the APIC cluster size.
Example:
POST
https://<IP address>/api/node/mo/uni/controller.xml
<infraClusterPol name='default' size=3/>
Step 2
Physically connect the APIC controllers that you want to add to the cluster.
Contracting the Cluster Examples
Contracting the APIC Cluster Using the GUI
Step 1
Step 2
Step 3
Step 4
Step 5
On the menu bar, choose SYSTEM > Controllers. In the Navigation pane, expand Controllers > apic_controller_name
> Cluster.
You must choose an apic_controller_name that is within the cluster and not the controller that is being decommissioned.
In the Work pane, the cluster details are displayed. This includes the current cluster target and current sizes, the
administrative, operational, and health states of each controller in the cluster.
Verify that the health state of the cluster is Fully Fit before you proceed with contracting the cluster.
In the Work pane, click Actions > Change Cluster Size.
In the Change Cluster Size dialog box, in the Target Cluster Administrative Size field, choose the target number to
which you want to contract the cluster. Click Submit.
Note
It is not acceptable to have a cluster size of two APIC controllers. A cluster of one, three, or more APIC controllers
is acceptable.
In the Work pane, in the Controllers area, choose the APIC that is last in the cluster.
Cisco APIC Troubleshooting Guide
150
Cisco APIC Cluster Management
Contracting the APIC Cluster Using the REST API
Example:
Step 6
Step 7
In a cluster of three, the last in the cluster is three as identified by the controller ID.
Click Actions > Decommission. The Confirmation dialog box displays. Click Yes.
The decommissioned controller displays Unregistered in the Operational State column. The controller is then taken
out of service and not visible in the Work pane any longer.
Repeat the earlier step to decommission the controllers one by one for all the APICs in the cluster in the appropriate
order of highest controller ID number to the lowest.
Note
The operation cluster size shrinks only after the last appliance is decommissioned, and not after the administrative
size is changed. Verify after each controller is decommissioned that the operational state of the controller is
unregistered, and the controller is no longer in service in the cluster.
You should be left with the remaining controllers in the APIC cluster that you desire.
Contracting the APIC Cluster Using the REST API
The cluster drives its actual size to the target size. If the target size is lower than the actual size, the cluster
size contracts.
Step 1
Set the target cluster size so as to contract the APIC cluster size.
Example:
POST
https://<IP address>/api/node/mo/uni/controller.xml
<infraClusterPol name='default' size=1/>
Step 2
Decommission APIC3 on APIC1 for cluster contraction.
Example:
POST
https://<IP address>/api/node/mo/topology/pod-1/node-1/av.xml
<infraWiNode id=3 adminSt='out-of-service'/>
Step 3
Decommission APIC2 on APIC1 for cluster contraction.
Example:
POST
https://<IP address>/api/node/mo/topology/pod-1/node-1/av.xml
<infraWiNode id=2 adminSt='out-of-service'/>
Cisco APIC Troubleshooting Guide
151
Cisco APIC Cluster Management
Commissioning and Decommissioning Cisco APIC Controllers
Commissioning and Decommissioning Cisco APIC Controllers
Commissioning a Cisco APIC Controller in the Cluster Using the GUI
Step 1
Step 2
From the menu bar, choose SYSTEM > Controllers.
In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.
Step 3
Step 5
From the Work pane, verify in the Active Controllers summary table that the cluster Health State is Fully Fit before
continuing.
From the Work pane, click the decommissioned controller that displaying Unregistered in the Operational State
column.
The controller is highlighted.
From the Work pane, click Actions > Commission.
Step 6
In the Confirmation dialog box, click Yes.
Step 7
Verify that the commissioned Cisco APIC controller is in the operational state and the health state is Fully Fit.
Step 4
Decommissioning a Cisco APIC Controller in the Cluster Using the GUI
Step 1
Step 2
On the menu bar, choose System > Controllers.
In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.
Step 3
In the Work pane, verify that the Health State in the Active Controllers summary table indicates the cluster is Fully
Fit before continuing.
In the Navigation pane, click an apic_controller_name that is within the cluster and not the controller that is being
decommissioned.
The controller details appear in the Work pane.
Step 4
Step 5
In the Work pane, click Actions > Decommission.
The Confirmation dialog box displays.
Step 6
Click Yes.
The decommissioned controller displays Unregistered in the Operational State column. The controller is then taken
out of service and no longer visible in the Work pane.
Note
• The operation cluster size shrinks only after the last appliance is decommissioned, and not after the
administrative size is changed. Verify after each controller is decommissioned that the operational state
of the controller is unregistered, and the controller is no longer in service in the cluster.
• After decommissioning the APIC controller, you must reboot the APIC for Layer 4 to Layer 7 services.
Reboot must be done before commissioning back the controller.
Cisco APIC Troubleshooting Guide
152
Cisco APIC Cluster Management
Replacing a Cisco APIC in a Cluster Using the CLI
Replacing a Cisco APIC in a Cluster Using the CLI
Note
• For more information about managing clusters, see Cluster Management Guidelines.
• When you replace an APIC, the password will always be synced from the cluster. When replacing
APIC 1, you will be asked for a password but it will be ignored in favor of the existing password in
the cluster. When replacing APIC 2 or 3, you will not be asked for a password.
Step 1
Step 2
Step 3
Step 4
Identify the APIC that you want to replace.
Decommission the APIC using the controller controller-id decommission command.
Note
Decommissioning the APIC removes the mapping between the APIC ID and Chassis ID. The new APIC typically
has a different APIC ID, so you must remove this mapping in order to add a new APIC to the cluster.
If you want to recommission the same APIC, follow these steps:
a) Restart the APIC using the acidiag reboot command.
b) Verify that the APIC boots without error.
c) Commission the APIC using the controller controller-id commission command.
d) Allow several minutes for the new APIC information to propagate to the rest of the cluster.
If you want to commission a new APIC, follow these steps:
a) Disconnect the APIC from the fabric.
b) Connect the replacement APIC to the fabric.
c) Commission the APIC using the controller controller-id commission command.
d) Boot the new APIC.
e) Allow several minutes for the new APIC information to propagate to the rest of the cluster.
Cisco APIC Troubleshooting Guide
153
Cisco APIC Cluster Management
Replacing a Cisco APIC in a Cluster Using the CLI
Cisco APIC Troubleshooting Guide
154
APPENDIX
E
Cisco APIC SSD Replacement
Use this procedure to replace the Solid-State Drive (SSD) in Cisco APIC.
• Replacing the Solid-State Drive in Cisco APIC, page 155
Replacing the Solid-State Drive in Cisco APIC
Step 1
Decommission the APIC.
a) On the menu bar, choose System > Controllers.
b) In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node. From this point,
select an apic_controller_name that is not the controller that is being decommissioned.
c) In the Work pane, verify that the Health State in the Active Controllers summary table indicates the cluster is Fully
Fit before continuing.
d) In the same Work pane, click Actions > Decommission after selecting the controller to be decommissioned..
e) Click Yes.
The decommissioned controller displays Unregistered in the Operational State column. The controller is then taken
out of service and no longer visible in the Work pane.
Step 2
This step is conditional and is required if your Cisco IMC version is earlier than 2.0(9c). The minimum Cisco IMC
version required to replace SSD is 2.0(9c). Upgrade to Cisco IMC version 2.0(9c) if required.
a) Use the Host Upgrade Utility (HUU) to upgrade to Cisco IMC version 2.0(9c).
b) Download the .iso image. See Cisco Host Upgrade Utility 2.0(7) User Guide to upgrade to Cisco IMC version
2.0(9c).
Step 3
This step is conditional and is required if you already have SSD installed. Perform this step before replacing an existing
SSD with a new SSD. This step is not required if you are installing a new SSD.
a) Log in to Cisco IMC.
b) Choose Storage > Physical Drive Info.
c) Select the SSD. From the Actions menu click Prepare for Removal.
Cisco APIC Troubleshooting Guide
155
Cisco APIC SSD Replacement
Replacing the Solid-State Drive in Cisco APIC
d) Click OK and select the SSD.
Step 4
Step 5
Physically remove the old SSD if any, and add the new SSD.
Create RAID volume using the newly installed SSD.
a) Log in to Cisco IMC.
b) Choose Storage > Physical Drive . Select the newly added physical drive.
c) Choose Storage > Controller Drive Info, and click Clear Foreign Config.
d) Click OK.
e) Choose StorageController Drive Info, and click Create Virtual Drive from Unused Physical Drives.
f) Select 0 from the Raid Level drop-down list.
g) Click Create Virtual Drive.
h) Select the newly created virtual drive and click Initialize.
i) Select the Initialize Type from the drop-down list and click Fast Initialize.
Step 6
Install the APIC image using the virtual media. In this step, the SSD is partitioned and the APIC software is installed on
the HDD.
a) Obtain the relevant APIC .iso image from CCO. The image version of the APIC must be the same version as
the other Controllers in the cluster.
b) Mount the APIC .iso image using the Cisco IMC vMedia functionality.
c) Boot or power cycle the controller.
d) During the boot process press F6 to select the Cisco vKVM-Mapped vDVD as the one-time boot device. You may
be required to enter the BIOS password. The default password is 'password'.
e) Follow the onscreen instructions to install the APIC software.
f) After the installation is completed, un-map the virtual media mount.
Step 7
Commission the APIC.
a) Select any other APIC that is part of the cluster. From the menu bar, choose SYSTEM > Controllers.
b) In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node. From this point,
select any apic_controller_name that is part of the cluster.
c) From the Work pane, click the decommissioned controller that displaying Unregistered in the Operational State
column.
d) From the Work pane, click Actions > Commission.
e) In the Confirmation dialog box, click Yes.
The commissioned controller displays the Health state as Fully-fit and the operational state as Available. The controller
should now be visible in the Work pane.
Cisco APIC Troubleshooting Guide
156
INDEX
(enabling) NX-OS Format 55
(enabling) Syslog 55
expanding cluster 145
exporting files 133
about 133
A
acidiag Command 125
ACL deny logging 23, 24
ACL permit and deny logs 25
ACL permit logging 20, 21, 22
atomic counters 30, 31, 32, 60, 67
about 60, 67
configuring 30
guidelines and restrictions 30
C
cluster 149, 150, 152
Commissioning 152
contracting 150
decommissioning 152
expanding 149
Contract permit logging 20, 21, 22
Contract permit logs 25
contracting cluster 146
core files 133
P
permit logging 25, 26
S
SNMP 41, 42, 43, 44
about 41
configuring policy 42
configuring trap destination 43
configuring trap source 44
SPAN 45, 46
about 45
configuring 46
guidelines and restrictions 46
syslog 52, 53
about 52
destination 53
source 53
T
D
digital optical monitoring 32
Digital Optical Monitoring 33, 35
DOM 32, 33, 35
E
endpoint connectivity 95
eventlog Command 68, 69, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82,
Taboo Contract deny logging 23, 24
Taboo Contract deny logs 25
Taboo Contract drop logging 23, 24
techsupport file 136
sending 136
techsupport files 133
traceroute 56
about 56
configuring 56
guidelines and restrictions 56
83, 84, 85, 87, 88, 89, 90
Cisco APIC Troubleshooting Guide
IN-1
Index
Cisco APIC Troubleshooting Guide
IN-2
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement