9 TROUBLESHOOTING

This chapter provides basic troubleshooting for diagnosing and fixing problems with VLAN, emulated LANs, PNNI links, and ATM traffic congestion.

9.1

TROUBLESHOOTING IP OVER ATM

You have configured an IP over ATM VLAN, but your network applications are not working. Use these questions and tests to help determine the cause of the problem.

1.

Check for connectivity: Try pinging between end nodes and from the ATM SmartSwitch (using ping

) to its end nodes. If you cannot ping, check physical connectivity (disconnected cable and so on).

2.

Check IP routes and addresses.

• Use the show route command to check the ATM SmartSwitch route table.

Are the destination addresses correct for the specified gateways?

Are there any routing loops?

Are one or more of the destination addresses mapped to the wrong subnet?

• Use show client

(ARP server is on the ATM SmartSwitch) to check the local client.

Does the client have the correct IP address?

Is the subnet correct? Is the ATM address correct?

Is the server type correct?

• Check end node configurations.

Are end nodes configured correctly?

3.

Check ARP statistics.

• Use show ipatmarp

(if the ARP server is on the ATM SmartSwitch).

Are there entries in the table?

Are the ATM addresses correct?

• Use show clientarp (ARP server is not on the SmartSwitch) to check local client’s ARP Table.

Are there entries in the table? If not, recheck client and end node configuration.

Are the ATM addresses correct?

4.

Check ILMI, UNI routes, and PVCs (if applicable).

• If using SVCs, use show ATMRoute

to check whether static UNI routes are correct and whether dynamic UNI routes are established and correct. If dynamic routes are incorrect or missing, try creating static routes instead.

• If using PVCs, use show pvc to check if PVCs connect the correct resources through the correct ports.

• If using PVCs, use show ipatmpvc

to check if local switch clients are mapped to the correct end node

IP addresses.

SmartSwitch ATM User Guide 9-1

Troubleshooting LAN Emulation Troubleshooting

5.

If working through these questions does not solve the problem, contact Cabletron Systems Customer

Service. (see Appendix B, "Technical Support").

9.2

TROUBLESHOOTING LAN EMULATION

You have configured an Emulated LAN and your network applications are not working. Use these questions and tests to help determine the cause of the problem.

1.

Check for connectivity. Try pinging between end nodes. Ping from the ATM SmartSwitch (using ping

) to its end nodes. If you cannot ping, check physical connectivity (disconnected cable and so on).

2.

Execute the show lecs command on the switch that contains the LECS. If the LECS is down, start it by executing the start lecs command.

If running distributed LANE services (LECS on one switch and LES and BUS on another switch) execute the show les

command on the switch running the LES and BUS. If the LES and BUS are down, start the LES and BUS by executing the start les

command.

3.

Check IP routes and addresses.

• Use show route command to check the ATM SmartSwitch route table.

Are the destination addresses correct for the specified gateways?

Are there any routing loops?

Are one or more of the destination addresses mapped to the wrong subnet?

• Use show client

to check the ATM SmartSwitch local ELAN client.

Does the client have the correct IP address?

Is the subnet correct?

Is the ATM address correct?

Is the server type correct?

• Check end nodes configurations.

Are end nodes configured correctly?

4.

If the ELAN spans multiple switches, check the following:

Is the LECS address correct on all switches?

Can all switches reach the switch providing LECS support?

If using the Well Known LECS Address, are all switches correctly mapped?

5.

Check the LECS database.

• Use show lecselan to check the names and numbers of ELANs.

Are ELAN names correct?

Is the ATM address of the LES correct?

6.

Check whether LES is connected.

• Use show lesclient

to check whether devices are registered with the LES. If clients are registered, check end node configuration. If not registered, check multi-point signaling.

• Use set leselan to turn off multi-point signaling on a per-ELAN basis.

9-2 SmartSwitch ATM User Guide

Troubleshooting Troubleshooting PNNI Links

Do devices begin to register with the LES and BUS once multi-point signaling is turned off?

7.

Check whether BUS is connected.

• Use show busclient to check whether devices are registered with the BUS. If clients are registered, check end node configuration. If not registered, check multi-point signaling.

• Use set leselan to turn off multi-point signaling on a per-ELAN basis.

Do devices begin to register with the LES and BUS once multi-point signaling is turned off?

• Check IISP routes to the switch containing the LES and BUS.

Are all IISP routes correct?

Does a new IISP route need to be added so devices can reach the LES and BUS?

8.

If working through these questions does not solve the problem, contact Cabletron Systems Customer

Service. (see Appendix B, "Technical Support").

9.3

TROUBLESHOOTING PNNI LINKS

You have physically connected another company’s ATM switch with your ATM SmartSwitch. Each switch supports

PNNI, but there is no connectivity between the two devices. When dealing with PNNI connectivity, two possible configurations must be considered:

• The ATM SmartSwitch and the other switch are in the same peer group

• The ATM SmartSwitch and the other switch are is different peer groups

Use the following procedures to diagnose and resolve PNNI connectivity problems.

9.3.1

Switches in Same Peer Group

1.

2.

3.

4.

Check the physical connection. Make sure that the switches are connected correctly.

Check that both switches are in the same peer group. On the ATM SmartSwitch, enter the show pnninode

command to view the peer group ID. If not the same peer group, perform the following:

Set the peer group ID on either switch to match the other. On the ATM SmartSwitch, use the set pnnipeergroup

command to change the peer group ID.

Check the signalling type of each switch. If either switch does not show PNNI as the signaling type on the connecting port. Perform the following:

Turn off ILMI and manually set the signaling type to PNNI. On the ATM SmartSwitch, enter the show portconfig

command to view signaling type for all ports. If necessary, use the set portconfig

command to turn off ILMI and manually set signaling to pnni10

.

If none of the above actions have corrected the problem, contact Cabletron Systems Customer

Service (see Appendix B, "Technical Support").

9.3.2

Switches in Different Peer Groups

1.

Check the physical connection between the peer groups. Make sure that the switches are connected correctly.

SmartSwitch ATM User Guide 9-3

Troubleshooting Congestion Troubleshooting

2.

3.

4.

5.

Make certain that the switches in the other peer group support multi-level PGLs and border nodes.

If not, the other switches must be placed in the same peer group as the ATM SmartSwitch if you want them to connect.

Are the switches within the peer groups communicating with each other? If not, fix the connectivity

problem within the peer group (see Section 9.3.1).

Has the Peer Group Leader (PGL) been elected in both groups? If not, start the election process. On the ATM SmartSwitch, use the set pnniplgelection command to start the PGL election process.

Do both peer groups have a parent node (grandparent node, great grandparent, etc.) in a common peer group?

If not, create a parent node within a higher-level peer group that’s common to both peer groups.

On the ATM SmartSwitch, use the add pnninode

command to create the parent node.

-

If they do, contact Cabletron Systems Customer Service (see Appendix B, "Technical Support")

9.4

TROUBLESHOOTING CONGESTION

If the bandwidth of your ATM SmartSwitch begins to decrease, and if connections are being lost or packets are being dropped at a high rate, it’s possible that your switch is becoming congested. Congestion can occur on the port level, the global switch level, or both levels.

If you suspect that your ATM SmartSwitch is experiencing congestion, follow the steps outlined below to diagnose and resolve the cause of congestion.

9.4.1

Diagnosing Congestion

1.

2.

3.

Enter the show portstats

command, and take the default of ( all

).

If cells are being dropped only on specific ports, proceed to the “Port Congestion” section.

If cells are being dropped on all ports, the indication is global congestion. Proceed to the “Global

Congestion” section.

9.4.2

Global Congestion

1.

Is the total cell drop rate equal to the Unknown VC cell drop rate?

• If yes, the switch is improperly set up. Check the switch configuration.

• If no, this indicates global congestion. Continue.

2.

Set the porttrafficcongestion

values to those recommended in the table below.

Table 9-1 Settings for Class of Service Queues

Service Class

CBR

CBR

Recommended Settings

Fewer than 100 connections on a port: Min = 64, Max = 1024

More than 100 connections on a port: Min = 128, Max = 1024

9-4 SmartSwitch ATM User Guide

Troubleshooting Troubleshooting Congestion

Table 9-1 Settings for Class of Service Queues (Continued)

Service Class Recommended Settings rt-VBR rt-VBR

Nrt-VBR

UBR

Bandwidth* utilization less than 20%: Min = 16, Max = 1024

Bandwidth* utilization greater than 20%: Min = 128, Max = 4096

Min = 256, Max = 4096

Min = 256, Max = 8192

ABR Min = 256, Max = 8192

*Use the show portconfig command to view bandwidth utilization

3.

Has the congestion subsided?

• If yes, you are done.

• If no, continue.

4.

Have you changed the EPD threshold ( set switchtrafficcongestion

command)?

• If yes, replace it to the default setting. If congestion subsides, you are done.

• If no, continue.

5.

Enter the show cacinfo

and show portconfig

commands for each port. Is the allocated bandwidth small and is the traffic mostly UBR?

• If no, go back to step 4 and check next port.

• If yes, continue.

6.

Enter the show porttrafficcongestion

command. Is the UBR queue

MaxValue

large?

• If no, go back to step 4.

• If yes, continue.

7.

Reduce the UBR queue

MaxValue

by a small amount, then wait a few minutes.

8.

Enter the show portstats

command, and take the default of all

. Is the number of cells dropped increasing for this port, and quickly decreasing for all other ports?

• If yes, proceed to the “Port Congestion” section.

• If no, continue.

9.

Is the number of cells being dropped by all other ports decreasing somewhat?

• If no, go back to step 6.

• If yes, continue.

10.

Enter the set caceqbwallocscheme command and set call admission control for this port to a more conservative policy ( moderate

or conservative

).

11.

Go back to step 4 until all ports have been checked.

9.4.3

Port Congestion

1.

Enter the show portstats

command a few times, noting the value for cells dropped and unknown

VCs dropped. Is the number of cells dropped equal to the number of VCs dropped?

SmartSwitch ATM User Guide 9-5

Events and Alarms Troubleshooting

• If yes, the switch is improperly set up. Check the switch configuration.

• If no, this indicates port congestion. Continue.

2.

3.

Enter the show cacinfo command for this port. Note the bandwidth allocated for each Quality of

Service on this port.

For each class of service, enter the set porttrafficcongestion

command. Set the

MaxValue

to the

value recommended in Table 9-1, “Setting for Class of Service Queues.”

4.

Have you performed step 3 for every class of service for this port?

• If no, go to step 3.

• If yes, continue.

5.

6.

Enter the set caceqbwallocscheme command for this port. Set call admission control for this port to a more conservative policy ( moderate or conservative ).

Check VC statistics for this port using either the show pvc /d

or show svc /d

command, whichever is appropriate. If the port belongs to the high virtual channel link (VCL), read the forward statistics.

If the port belongs to the low VCL, read the backward statistics. If the port belongs to both high and low VCLs, read both statistics.

7.

Is the number of cells received increasing?

• If no, go through step 6 a few more times. If cells received still do not increase and congestion persists, contact Cabletron Customer support.

• If yes, continue.

• Enter the show cacinfo

command for this port. Is the Allocated Bandwidth less than the Cell

Reception Rate obtained from show pvc /d

or show svc /d in step 6?

• If no, go through step 6 a few more times. If cells received still do not increase and congestion persists, contact Cabletron Customer support.

• If yes, this VC is misbehaving. Take appropriate action, for example, terminate the VC.

9.5

EVENTS AND ALARMS

ATM SmartSwitches record and report their operation in real-time through the use of events and alarms. An event is an occurrence of a significant activity. For instance, a port going down or a client joining an ELAN are examples of events. Alarms are a specific class of events defined as “events that the user needs to know about or attend to immediately.” Alarms do not always indicate switch faults. Alarms may also be informational events. For instance,

“LECS Operational” is an example of an alarm that is not a switch fault, but is an activity that the user should know about immediately.

9.5.1

Event Categories

Events are grouped into the following categories:

• Critical — Impacts the entire switch, leaving the system unavailable or in a degraded state

• Major — Impacts a feature of the switch, leaving the feature unavailable or in a degraded state

• Minor — Impacts the system or feature, leaving it in a sub-optimal state

• Informational — An occurrence of an activity that the user should know about

9-6 SmartSwitch ATM User Guide

Troubleshooting Events and Alarms

Both events and alarms are stored within circular memory buffers. When the buffers become full, older events and alarms are overwritten by newer entries. Both events and alarms are stored in shared RAM. However, the 40 most recent alarms are also stored in flash RAM. Storing these 40 alarms in flash RAM makes them persistent between reboots of the ATM SmartSwitch, and provides information about the state of the switch prior to reboot.

Note Alarms are collected and stored in flash RAM in groups of four. As a result, some of the most recent alarms may not be persistent. For example, there are 24 (6 times

4) alarms stored in flash RAM. If a 25th alarm occurs, and the switch is rebooted, only the 24 alarms are persistent. The 25th alarm is dropped because the number of alarms (after 24) did not reached the next multiple of four (28).

9.5.2

Viewing Events and Alarms

Use the show events

command to view a list of the currently logged events. For example,

SmartSwitch # show events

Index(ALL) :

0 33554474 MAJOR EVENT 000:00:04:311

---------------------------------------------------

LES ReadServerConfig: Unable to open config file les.db

1 33554653 INFO EVENT 000:00:04:320

---------------------------------------------------

LECS Database non existing - creating default ELAN

2 117571585 MINOR EVENT 000:00:07:341

---------------------------------------------------

SAAL connection has become active, initiated locally

Port ID 0x01c41000

Protocol 0x02

3 117571585 MINOR EVENT 000:00:07:585

---------------------------------------------------

SAAL connection has become active, initiated locally

More(<space>/q)?:

Events are displayed in the following format:

• Event number — The index number of the event in the circular buffer

• Event ID — A unique ID assigned to the event

• Category — Whether this event is critical, major, minor, or informational

• Time — Time of event, in switch up-time in hours, minutes, seconds, and milliseconds

• Object — The object affected by the event (port, LEC, and so on)

• Description — Brief message describing the event

Event messages can be automatically displayed on the ATM SmartSwitch console. Use the set eventdisplay command to display events on the console as they occur:

SmartSwitch # set eventdisplay

EventDisplay(OFF) : on

SmartSwitch #

SmartSwitch ATM User Guide 9-7

Events and Alarms Troubleshooting

Note Depending on the activity of your ATM SmartSwitch, the appearance of events on the ATM SmartSwitch may be too frequent to use the console comfortably. It is recommended that you turn on the automatic display of events only when troubleshooting.

Use the show alarms command to view a list of the currently logged alarms. For example,

SmartSwitch # show alarms

Index(ALL) :

0 33554702 000:07:05:300

--------------------------------------------------pvcm_cac_admit: failed 501037

1 33554652 023:56:23:317

---------------------------------------------------

LECS Operational

2 117506049 024:01:54:083

---------------------------------------------------

Failed to re-establish SAAL connection

Port ID 0x01c81000

T309 10000

3 117506049 024:01:54:430

---------------------------------------------------

More(<space>/q)?:

Alarms are displayed in the following format:

• Alarm number — The index number of the alarm in the circular buffer

• Alarm ID — A unique ID assigned to the alarm

• Time — Time of alarm, in switch up-time in hours, minutes, seconds, and milliseconds

• Object — The object affected by the alarm (port, LEC, and so on)

Alarm messages can be automatically displayed on the ATM SmartSwitch console. Use the set alarmdisplay command to display alarms on the console as they occur:

SmartSwitch # set alarmdisplay alarmDisplay(OFF) : on

SmartSwitch #

9.5.3

Deleting Events and Alarms

To delete events or alarms currently logged within your ATM SmartSwitch, use the delete events

and delete alarms commands, respectively.

9-8 SmartSwitch ATM User Guide

Troubleshooting Saving Core Dumps

9.6

SAVING CORE DUMPS

The ATM SmartSwitch core dump feature allows you to specify a local Ethernet host where, in the event of a system failure, the ATM SmartSwitch sends a copy of its memory. ATM SmartSwitch system memory is saved to two files, one containing CPU memory ( core_cpu ), the other common memory ( core_cmn ). These files can then be sent to

Cabletron customer support for analysis.

Note To use the core dump feature, the local Ethernet host must be running TFTP server software, and you must have write access to the TFTP directory.

Enter the set coredump

command to enable the core dump feature. For example,

SmartSwitch # set coredump

EnableCoreDump(n) : y — “y” to enable core dump feature

ServerIP() : 204.95.77.240 — IP address of my TFTP server

CoreDumpFile() : /tftpboot/bobr/core — full path name for core dump files userName() : bobr — login name on the server

UserPassword() : — password

SmartSwitch #

Note The set coredump

command uses FTP to create the core_cpu and core_cmn files. If your server does not run FTP, create these files manually. Then execute the set coredump

command.

Note On UNIX systems, make sure that the permissions are set correctly so that data can be written.

Note For security, the set coredump

command retains your password only long enough to create the core dump files. Your password is then dropped from system memory.

To see the current core dump configuration, enter the show coredump

command.

SmartSwitch # show coredump

Core Dump Enabled : Yes

Core Dump Server IP : 204.95.77.240

Core Dump File : /tftpboot/bobr/core

SmartSwitch #

SmartSwitch ATM User Guide 9-9

Saving Core Dumps Troubleshooting

If a system failure occurs while the core dump feature is enabled, the ATM SmartSwitch console appears similar to the example below. The ATM SmartSwitch then begins sending images of its memory to the core dump files on the TFTP server.

Illegal access. Bus Error.

IP: e0103288 PFP: e04be080 r0(pfp): e04be040 r1(sp): e04be0c0 r2(rip): e00dd7dc r3 : 00000000 r4 : e00f8f0c r5 : e0409f10 r6 : 00000003 r7 : e00f8f0c r8 : e0409f40 r9 : 00000003 r10 : 00000030 r11 : e00f8f0f r12 : 00000008 r13 : 00000001 r14 : e00d22f0 r15 : 00000008 d2000000: Core Dump

Common DRAM dumped to /tftpboot/bobr/core_cmn

CPU DRAM dumped to /tftpboot/bobr/core_cpu ffffffff ffffffff ffffffff ffffffff *................* d2000010: ffffffff ffffffff ffffffff ffffffff *................* d2000020: ffffffff ffffffff ffffffff ffffffff *................* d2000030: ffffffff ffffffff ffffffff ffffffff *................* d2000040: ffffffff ffffffff ffffffff ffffffff *................* d2000050: ffffffff ffffffff ffffffff ffffffff *................* d2000060: ffffffff ffffffff ffffffff ffffffff *................* d2000070: ffffffff ffffffff ffffffff ffffffff *................* d2000080: ffffffff ffffffff ffffffff ffffffff *................* d2000090: ffff

SmartSwitch Start-up Code

Cabletron Systems Inc.

Copy the information displayed on the console and send it to your Cabletron customer support representative along

with the core dump files. (See Appendix B, "Technical Support")

9-10 SmartSwitch ATM User Guide