Avaya Syslog Implementation Guide

Avaya Syslog Implementation Guide
ABSTRACT
This document provides implementation guidelines to add and maintain logging services on Avaya
platforms. Configurations and recommendations are given for several platforms including:
Communication Manager 4.0-5.x, G450 Gateway, IG550 Gateway, Cruiser board, Crossfire board, and
SES. Configuration and recommendations are also supplied for the 4600 and 9600 series phones.
Information is also provided regarding the impact of syslog traffic on network bandwidth.
The intent of this document is to provide training on logging, and to provide guidelines for
implementing Avaya solutions. It is intended to supplement the product documentation, not replace it.
May 2008
COMPAS ID 136974
Avaya Syslog Implementation Guide
All information in this document is subject to change without notice. Although the information is
believed to be accurate, it is provided without guarantee of complete accuracy and without warranty of
any kind. It is the user’s responsibility to verify and test all information in this document. Avaya shall not
be liable for any adverse outcomes resulting from the application of this document; the user accepts full
responsibility.
© 2008 Avaya Inc. All Rights Reserved.
Avaya and the Avaya Logo are trademarks of Avaya Inc. or Avaya ECS Ltd., a wholly owned subsidiary
of Avaya Inc. and may be registered in the US and other jurisdictions. All trademarks identified by ® and
™ are registered trademarks or trademarks, respectively, of Avaya Inc. All other registered trademarks or
trademarks are property of their respective owners.
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
1
1.
OVERVIEW OF LOGGING ............................................................................................................................5
1.1.
SYSLOG BASICS ...........................................................................................................................................6
1.2.
LOGGING LEVELS ........................................................................................................................................7
1.3.
LOGGING FACILITIES ...................................................................................................................................8
1.4.
SECURITY DESIGN CONSIDERATIONS ..........................................................................................................9
1.5.
SYSLOG MESSAGES .....................................................................................................................................9
1.5.1. The PRI Section .....................................................................................................................................9
1.5.2. The Header Section.............................................................................................................................. 10
1.5.3. The Message Section............................................................................................................................ 11
2.
EVENT LOGGING IN CM 3.0 AND LATER .............................................................................................. 12
2.1.
ACCESSING SYSTEM LOGS THROUGH THE WEB INTERFACE ....................................................................... 12
2.2.
SELECT LOG TYPES ................................................................................................................................... 14
2.2.1. Logmanager debug trace ..................................................................................................................... 14
2.2.2. Operating system boot messages ......................................................................................................... 15
2.2.3. Linux scheduled task log (CRON)........................................................................................................ 16
2.2.4. Linux syslog ......................................................................................................................................... 17
2.2.5. Linux access security log ..................................................................................................................... 18
2.2.6. Linux login/logout/reboot log .............................................................................................................. 20
2.2.7. Linux file transfer log .......................................................................................................................... 20
2.2.8. Watchdog logs ..................................................................................................................................... 20
2.2.9. Platform command history log............................................................................................................. 21
2.2.10.
HTTP/web server error log ............................................................................................................. 22
2.2.11.
HTTP/web SSL request log ............................................................................................................. 22
2.2.12.
Communication Manager Restart log ............................................................................................. 23
2.3.
SELECT A VIEW ......................................................................................................................................... 24
2.3.1. IP events .............................................................................................................................................. 24
2.3.2. Platform bash command history log .................................................................................................... 25
2.3.3. Communication Manager’s raw Message Sequence Trace (MST) ...................................................... 25
2.3.4. Communication Manager’s processed Message Tracer (MDF).......................................................... 25
2.3.5. Communication Manager’s interpreted Message Tracer (MTA) ........................................................ 25
2.3.6. Communication Manager’s hardware error and alarm events ........................................................... 25
2.3.7. Communication Manager’s software events ........................................................................................ 26
2.4.
SELECT EVENT RANGE .............................................................................................................................. 26
2.5.
DISPLAY FORMAT ..................................................................................................................................... 26
2.6.
INTERPRETING THE LOG ENTRIES ............................................................................................................... 26
2.7.
ADVANCED CM 3.0 SERVER LOG MANAGEMENT ..................................................................................... 28
2.8.
CONFIGURING SYSLOG VIA LINUX CLI ..................................................................................................... 28
2.9.
VIEWING AND FILTERING LOG FILES......................................................................................................... 31
2.9.1. The logv, logc, logw Commands .......................................................................................................... 31
2.9.2. Filtering and Searching With logv ....................................................................................................... 32
2.9.3. Using logc and logv ............................................................................................................................. 33
2.9.4. Using logc ............................................................................................................................................ 33
2.9.5. Using logw ........................................................................................................................................... 34
2.10.
DOWNLOADING LOG DATA (INTERNAL) ................................................................................................... 36
2.11.
LOG FILE DETAILS (INTERNAL) ................................................................................................................. 38
2.12.
USEFUL ECS LOG CONTENT (INTERNAL) .................................................................................................. 38
2.12.1.
Significant Entries (Internal) .......................................................................................................... 39
2.12.2.
Miscellaneous Contents .................................................................................................................. 39
2.12.3.
Linux syslogd Logs .......................................................................................................................... 40
2.12.4.
Linux Access & Security Logs ......................................................................................................... 40
2.12.5.
Linux Kernel Messages ................................................................................................................... 41
2.12.6.
Watchdog Logs................................................................................................................................ 41
2.12.7.
Platform Command Logs ................................................................................................................ 41
2.12.8.
HTTP/Web Server Logs .................................................................................................................. 42
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
2
2.12.9.
2.12.10.
3.
Linux cron Logs .............................................................................................................................. 42
ACM Restart Log** ........................................................................................................................ 43
UPDATES AND CHANGES IN CM 4.0 LOGGING ................................................................................... 44
3.1.
SYSLOG FILE CONSOLIDATION .................................................................................................................. 45
3.2.
ENHANCEMENTS TO LOGGING COMMAND HISTORY ................................................................................. 46
3.2.1. BASH Command Logging .................................................................................................................... 47
3.2.2. Web Interface Activity .......................................................................................................................... 48
3.2.3. Logging CMS Activity .......................................................................................................................... 49
3.3.
HOW A CONFIGURED /ETC/SYSLOG.CONF SHOULD LOOK ......................................................................... 50
4.
CONFIGURING LOGGING IN CM 4.0 ....................................................................................................... 51
4.1.
CONFIGURING SYSLOGGING ON CM 4.0 AND LATER ................................................................................. 51
4.1.1. Setting the Logging Levels .................................................................................................................. 52
4.1.2. Which Log Data Can Be Sent? ............................................................................................................ 53
4.1.3. How To Configure An External Linux Server To Receive Logs ........................................................... 53
5.
TROUBLESHOOTING SYSLOG ................................................................................................................. 55
5.1.
5.2.
5.3.
6.
FIREWALL ................................................................................................................................................. 55
REVIEW THE SYSLOG.CONF FILE ............................................................................................................... 57
CHECK THE SYSLOGD SERVICE .................................................................................................................. 59
EVENT LOGGING FOR 4600 AND 9600 SERIES TELEPHONES.......................................................... 60
6.1.
6.2.
7.
CONFIGURATION ....................................................................................................................................... 60
ENABLING/DISABLING LOGGING ON ENDPOINTS ...................................................................................... 61
GATEWAY LOGGING .................................................................................................................................. 62
7.1.
HIGH LEVEL FEATURE DESCRIPTION......................................................................................................... 62
7.2.
MEDIA GATEWAY SEVERITIES .................................................................................................................. 62
7.3.
MEDIA GATEWAY SYSLOG MESSAGE FORMAT ......................................................................................... 64
7.4.
MEDIA GATEWAY SYSLOG FILTERING ...................................................................................................... 65
7.4.1. Architectural Overview (Internal) ....................................................................................................... 65
7.4.2. Interpretation of Syslog server entries from a media gateway ............................................................ 67
7.4.3. Provisioning and Installation Description ........................................................................................... 69
7.4.4. CPU / Memory / Bandwidth / Resource Needs .................................................................................... 69
7.5.
BRANCH GATEWAY G450 SYSLOG CONFIGURATION ................................................................................ 70
7.5.1. G450 Syslog Example .......................................................................................................................... 72
8.
TN BOARD SYSLOGGING (INTERNAL) .................................................................................................. 74
8.1.
CM 5.1 AND LATER SYSTEM-WIDE SETTINGS ........................................................................................... 74
8.2.
SYSLOG ON CLAN (TN799) AND CROSSFIRE (TN2602) BOARDS ............................................................ 75
8.3.
SYSLOG ON IPSI (TN2312) BOARDS (INTERNAL) ..................................................................................... 76
8.4.
ENABLING SYSLOG WITH DUPLEX CROSSFIRE (TN2602) BOARDS............................................................ 77
8.5.
CHANGING SYSLOG FEATURES WITH THE TN BOARD COMMAND LINE ................................................... 78
8.6.
LOGGING INTO TN BOARDS ...................................................................................................................... 79
8.6.1. IPSI Board ........................................................................................................................................... 79
8.6.2. Logging into CLAN and Crossfire Boards ........................................................................................... 80
8.7.
IPSI DEBUG LOGGING AND LEVELS .......................................................................................................... 81
8.7.1. IPSI Objects and Levels ....................................................................................................................... 81
8.8.
FIRMWARE LEVELS FOR TN BOARD SYSLOGGING .................................................................................... 82
8.9.
CAVEATS FOR TN BOARD LOGGING ......................................................................................................... 82
9.
ENABLING SYSLOG IN SIP ENABLEMENT SERVER (SES)................................................................ 83
10.
SYSLOG AND CMS ........................................................................................................................................ 84
11.
SYSLOG AND INTUITY AUDIX .................................................................................................................. 84
12.
SYSLOG AND MODULAR MESSAGING................................................................................................... 85
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
3
12.1.
SYSLOG CONFIGURATION .......................................................................................................................... 85
12.1.1.
Events to Send to Syslog.................................................................................................................. 86
12.1.2.
Adding the Syslog Server ................................................................................................................ 86
13.
SYSLOG AND THE AUDIO CODES GATEWAY..................................................................................... 87
13.1.
EXTERNAL SYSLOG ................................................................................................................................... 87
13.1.1.
Logging Levels Configuration......................................................................................................... 87
13.2.
INTERNAL LOGGING .................................................................................................................................. 89
14.
COMMON TROUBLESHOOTING SCENARIOS ...................................................................................... 90
14.1.
14.2.
14.3.
15.
IPSI BOARD LINK FAILURE ....................................................................................................................... 90
NETWORK TIME PROTOCOL BLOCKED TO IPSI BOARD ............................................................................. 91
IPSI BOARD DUPLEX MISMATCHES .......................................................................................................... 93
SYSLOG BANDWIDTH UTILIZATION ..................................................................................................... 94
15.1.
BANDWIDTH UTILIZATION IN A LOADED ENVIRONMENT .......................................................................... 96
16.
SYSLOG AND TRAFFIC ENGINEERING ................................................................................................. 97
17.
LOGGING WITH KIWI TOOLS .................................................................................................................. 99
17.1.
17.2.
17.3.
17.4.
BTJ
AR RS
COLLECTING LOGS .................................................................................................................................... 99
VIEWING AND SENDING LOG FILES ......................................................................................................... 102
RULES AND FILTERING WITH KIWI SYSLOG ............................................................................................ 104
HIGHLIGHTING TEXT WITHIN KIWI ......................................................................................................... 109
Avaya Syslog Implementation Guide
TRS
4
1. Overview of Logging Logging via Syslog is a way of sending system information to a common collection site via TCP/IP. This information can then be analyzed to pinpoint system failures, security breaches, or to analyze specific system events. The Syslog application was invented and written by Eric Allman in the 1980’s and was originally designed to provide logging capabilities for the sendmail program. In 2001, Syslog was standardized as IETF RFC 3164. As systems progressed in complexity, a logging mechanism was in more demand and syslog’s popularity increased. The ability to log events provides utility in several areas. Logging can be used to benchmark new applications, so that faults are more easily detected in the future. Logging can also be used to troubleshoot existing applications; indeed, this is one of the first places that experienced engineers look if a problem arises. Applications are usually designed to provide some clues as to their overall health. These messages help engineers to understand how the system is operating, or if something is wrong. There is rarely a uniform method of event logging in application software. Some applications are designed to provide logging through built‐in logging applications; others provide logging via the Syslog application. The syslog application is designed to take messages from multiple applications or devices and write them to a single location. Logging can be local or remote. Most systems can be set up to log messages to the system itself (local, or log them to a syslog server residing at a different location (remote). Many Avaya devices support the Syslog application. These systems include: CM, S8xxx series, all branch gateways (except G700), CLAN boards, Crossfire boards, H.323/SIP endpoints, and IPSI’s. The following example illustrates why logging is important. A customer as multiple gateways attached to a CM server with about more than 1,000 endpoints. The CM server reboots resulting in some gateways re‐registering to the CM server, while others remained registered to their respective LSP’s. Rather than log into each gateway individually to determine the problem, it would be more beneficial to have logging turned on in each device. These devices could then send their logs to a centralized syslog server where they could be quickly analyzed. The logs from different sources can be color coded on the syslog server and names assigned to ip addresses (e.g 10.1.4.30 – CLAN 1 etc). Engineers can then easily search for a specific event in the log file to see what is different from a phone or gateway that went to the main CM server compared to a device that went to the LSP. Also, since the Syslogging server can have a large hard drive (500 GB), the chances of the logs wrapping around are minimized. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
5
1.1.
Syslog Basics Almost any device with large storage capacity can be used as a syslog server (log collector). This includes an engineer’s laptop. A free tool that can be downloaded and used for syslogging is available from Kiwi Tools, http://www.kiwisyslog.com. Once the software is loaded, simply start the syslogging daemon, and point the devices that will send logs to the IP address of the laptop. While using a laptop as a log collector makes sense in situations where logging has not been set up by the customer, a much more robust and usable scenario is to have remote devices log to a centralized log collector as illustrated below. If bandwidth is a major concern, a distributed logging architecture can be implemented. This architecture involves strategically placing log collectors on individual LAN’s, thus minimizing the need for logging data to traverse busy wide area network links . Figure 1 E SD GROUND JACK
POW ER
1
2
3
4
5
6
7
8
9
10
11
12
13
14
POWE R
FAN OR POWE R FAIL
FAN AND POWE R OK
AC INPUT
DC INPUT
A CTIVE RING
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
6
1.2.
Logging Levels Event messages can be logged at several levels. The amount logged depends upon the needs of the customer and troubleshooting engineer. Logging too much data takes up needless disk space and makes it difficult to parse through and find the actual problem. Logging too little, may not provide enough data to determine if the system is healthy, or troubleshoot a known issue. Below is a list of the logging levels that are available for syslog capable applications. It is suggested that logging levels be initially be set to log most of the major conditions that can impact a system conserve network and disk resources. For example, logging levels 0‐3 may be sufficient for administrators to determine major events to a system. It will most likely take some experimentation to get the logging levels just right. During major event, logging can be adjusted to be more granular to aid in the troubleshooting process. Once set, the logging levels may still need temporary adjustment during the troubleshooting process. Numerical Code Severity 0 Emergency: System is unusable
1 Alert: Action must be taken immediately
2 Critical: Critical conditions 3 Error: Error conditions 4 Warning: Warning conditions 5 Notice: Normal but significant condition
6 Informational: Informational messages
7 Debug: debug‐level messages BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
7
1.3.
Logging Facilities The logging facility helps the syslog server determine where the message originated from. There are 24 facilities available for use. These facilities are not application dependent, but they are system specific. For example, logging messages from a mail server should be sent to facility 2, while other messages about start up or shutdown events may be sent to a different facility. Below is a chart of the various logging facilities specified in RFC 3164 and the associated numerical values. Note that Local 0 through Local 7 can be customized to suit customer needs. For example, a customer may be currently using Local 7 for network related events; therefore a different local may be used for IPSI events. Numerical Code Facility
0 Kernel messages 1 User‐level messages 2 Mail system
3 System daemons 4 Security/authorization messages 5 Messages generated internally by syslogd 6 Line printer subsystem 7 Network news subsystem 8 UUCP subsystem 9 Clock daemon 10 Security/authorization messages 11 FTP daemon
12 NTP subsystem 13 Log audit
14 Log alert
15 Clock daemon 16 Local use 0
`17 Local use 1
18 Local use 2
19 Local use 3 (Used by CM) 20 Local use 4
21 Local use 5
22 Local use 6
23 Local use 7 (Used by IPSI) BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
8
1.4.
Security Design Considerations Many syslog daemons will not provide any sort of authentication for incoming syslog messages. This is an obvious security concern amplified by the following RFC 1364 statement: The payload of any IP packet that has a UDP destination port of 514 MUST be treated as a syslog message. This treatment allows an attacker to send messages that a syslog server is not meant to receive. The attacker can alter those messages to create a state of confusion for troubleshooting engineers. Additionally, there is a risk that an attacker can mount a denial of service attack on the syslog server by sending more message traffic than the server or network is able to handle. All of this can easily be accomplished with software tools available on the internet. These types of attacks can be mitigated by proper network segmentation and by implementing a firewall that separates the devices and collector. The firewall allows only connections from known sources and destinations, lessening the likelihood of this type of attack. Additionally, a different port can be chosen for syslog traffic other than port 514. Once a different port is being used, port 514 should then be blocked by a firewall or access list. 1.5.
Syslog Messages The syslog message itself is broken down into three sections, the PRI, the header, and the message. The overall length of the message cannot exceed 1,024 characters. Below is the structure of a syslog message as specified in RFC 3164. [PRI] [ Header ] [ Message ] [PRI] [ (timestamp ) (host) ] [ (tag ) ( message )] 1.5.1. The PRI Section [PRI] [ Header ] [ Message ] [PRI] [ (timestamp ) (host) ] [ (tag ) ( message )] The primary or PRI field contains a number representing the facility and the priority of the message. According to RFC 3164: The priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the severity. Therefore the PRI field can be represented as: (Facility * 8 + Severity). The table below is designed to make converting the PRI value into facility and severity level easier. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
9
Facility Level Kernel User Mail System Security Syslog(d) Line printer Network news UUCP Clock Security FTPd NTPd Log audit Log alert Clock daemon Local 0 Local 1 Local 2 Local 3 Local 4 Local 5 Local 6 Local 7 Emerg 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Alert 0 0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 1 1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 Severity Critical 2 2 10 18 26 34 42 50 58 66 74 82 90 98 106 114 122 130 138 146 154 162 170 178 186 Error Warn 3 3 11 19 27 35 43 51 59 67 75 83 91 99 107 115 123 131 139 147 155 163 171 179 187 Notice 4 4 12 20 28 36 44 52 60 68 76 84 92 100 108 116 124 132 140 148 156 164 172 180 188 5 5 13 21 29 37 45 53 61 69 77 85 93 101 109 117 125 133 141 149 157 165 173 181 189 Info Debug 6 6 14 22 30 38 46 54 62 70 78 86 94 102 110 118 126 134 142 150 158 166 174 182 190 1.5.2. The Header Section [PRI] [ Header ] [ Message [PRI] [ (timestamp ) (host) ] [ (tag ) ( message )] ] The header consists of two fields, a timestamp and a hostname. The timestamp field represents the time that an event occurred local to that device. This is an important concept since several devices could be deployed in multiple time zones, with the syslog collection server in yet another time zone. The hostname field represents the local name of the device, but is not the fully qualified domain name. For example, device4 would appear in the syslog message as device4, not device4.localhost.local domain. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
10
7 7 15 23 31 39 47 55 63 71 79 87 95 103 111 119 127 135 143 151 159 167 175 183 191 1.5.3. The Message Section [PRI] [ Header ] [ Message ] [PRI] [ (timestamp ) (host) ] [ (tag ) ( message )] The message part of a syslog entry contains two fields, the tag and the message portion. The tag field contains the name of the program or process that created the message. The message field is a freeform text message describing the problem or event. It is important to note that all syslogging software is not created equally. A syslog message from RFC3164 compliant software will look similar to the one below: <133>Apr.22.13:12:23.device1.logger:.message.with SDSC.syslog.using.UDP This message has all the components specified by the RFC, PRI, Header, and Message complete with the required fields. Red Hat 8.0, however, by default will produce a message similar to the one below: <133>logger.message.with RedHat8.syslog.using.UDP This message does have a PRI, but the header is missing. The message including the tag is present. This difference can be critically important since there is no timestamp available. Therefore care should be taken when choosing a logging application. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
11
2. Event Logging in CM 3.0 and Later 2.1.
Accessing system logs through the Web interface Prior to Communications Manager Release 4.0, event log viewing is only available locally on the Communications Manager server. This section will show how to view and interpret the various logs that are available to administrators. Details on setting up and analyzing logging on Communications Manager 3.0 via the Linux command line will be discussed throughout this section. Click on the Launch Maintenance Web Interface link. The Integrated Management: Maintenance Web Pages license agreement and the navigation pane display. Select Diagnostics > System Logs from the left‐side navigation pane. The System Logs page displays BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
12
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
13
2.2.
Select Log Types This section of the System Logs page lists several logs and their contents: ● Logmanager debug trace ● Operating system boot messages ● Linux scheduled task log (CRON) ● Linux syslog ● Linux access security log ● Linux login/logout/reboot log ● Linux file transfer log ● Watchdog logs ● Platform command history log ● HTTP/web server error log ● HTTP/web SSL request log ● Communication Manager Restart log Note:: If more than one log is selected, the output is merged and displayed chronologically. If the merged log view is selected, the user can always tell from which log the entry was written by looking at the log‐name field on the entry. This field follows the sequence number field, immediately after the timestamp, and is separated by colons (see also Interpreting the log entries section). 2.2.1. Logmanager debug trace This log lists: ● All entries contained in the Avaya Communication Manager list history report. ● IP events: use "IPEVT" in the Match Pattern field or select the appropriate view (see IPevents section for more information). ● Auto trace‐route commands (a subset of the IPEVT entries) ● Process entries such as restarts, initializations, shutdowns, duplication status, process errors, system alarms, and communication with external gateways and port networks. Tip: As an example to restrict/sort the log report type [action (use only the left bracket) in the Match Pattern field and check the Match Pattern box to view the list history‐like entries. Use this for archiving and retrieving list history legacy reports that are deleted under certain circumstances. Using only “action” in the Match Pattern field displays all of list history entries plus additional debug entries. To export the log to a separate file, there are two options: ● Select View > Source in the IE browser menu or right‐click in the report pane. ● Copy and paste to a text processing application. Tip: BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
14
2.2.2. Operating system boot messages This log lists the boot‐up processes from the operating system. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
15
2.2.3. Linux scheduled task log (CRON) This log lists scheduled Linux processes. Use the Web interface to schedule backups. Note: Backups and Restores are the only scheduled process that can be initiated from the Web interface. The figure below shows two hourly cleanup cycles from a sample CRON log. Note: 20041109:230101000:6084:lxcron:MED:server_name
cron.hourly)
20041109:230001000:6083:lxcron:MED:server_name
1)
20041109:230000000:6082:lxcron:MED:server_name
sess_cleanup)
20041109:225000000:6081:lxcron:MED:server_name
1)
20041109:224000000:6080:lxcron:MED:server_name
1)
20041109:223000000:6079:lxcron:MED:server_name
1)
20041109:222000000:6078:lxcron:MED:server_name
1)
20041109:221000000:6077:lxcron:MED:server_name
1)
20041109:220100000:6076:lxcron:MED:server_name
cron.hourly)
20041109:220000000:6075:lxcron:MED:server_name
1)
20041109:220000000:6074:lxcron:MED:server_name
sess_cleanup)
20041109:215000000:6073:lxcron:MED:server_name
1)
20041109:214000000:6072:lxcron:MED:server_name
1)
20041109:213000000:6071:lxcron:MED:server_name
1)
20041109:212000000:6070:lxcron:MED:server_name
1)
20041109:211000000:6069:lxcron:MED:server_name
BTJ
AR RS
CROND[4375]: (root) CMD (run-parts /etc/
CROND[4372]: (root) CMD (/usr/lib/sa/sa1 1
CROND[4371]: (root) CMD (/opt/ecs/sbin/
CROND[1593]: (root) CMD (/usr/lib/sa/sa1 1
CROND[31856]: (root) CMD (/usr/lib/sa/sa1 1
CROND[29163]: (root) CMD (/usr/lib/sa/sa1 1
CROND[26454]: (root) CMD (/usr/lib/sa/sa1 1
CROND[24283]: (root) CMD (/usr/lib/sa/sa1 1
CROND[21591]: (root) CMD (run-parts /etc/
CROND[21424]: (root) CMD (/usr/lib/sa/sa1 1
CROND[21423]: (root) CMD (/opt/ecs/sbin/
CROND[18662]: (root) CMD (/usr/lib/sa/sa1 1
CROND[15900]: (root) CMD (/usr/lib/sa/sa1 1
CROND[13742]: (root) CMD (/usr/lib/sa/sa1 1
CROND[11032]: (root) CMD (/usr/lib/sa/sa1 1
CROND[8323]: (root) CMD (/usr/lib/sa/sa1 1
Avaya Syslog Implementation Guide
TRS
16
2.2.4. Linux syslog This log lists ● Linux process (system) messages ● Server (Linux platform) errors (in an un‐interpreted format) Note:
To view Communication Manager, Linux alarms, and other hardware errors use the Alarms > Current Alarms from the Maintenance Web Interface for a clearer view of the application and platform alarms, respectively. Using the Web interface report also identifies which errors require attention. Communication Manager errors are not logged in the Linux syslog but appear in the Logmanager debug trace log. Items logged: ● Linux operating system restarts ● Tripwire integrity checks (look for “…twd” entries) ● Disk problems ● Normal events ● Save translations The Server Maintenance Engine and Global Maintenance Manager processes monitor this log and report alarms. Note BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
17
2.2.5. Linux access security log This log lists: ● Successful and rejected logins/logoffs from either the Web interface or SAT. Note: This log does not report access or changes to the Web interface; these appear in the HTTP error log. Any activity at the Web interface appears in this log but not in Communication Manager’s list history report. ● At the first incorrect login, the log entry reads “…LOGIN_LOCKOUT…probation interval for login [login] begins,” indicating that a timer has started. ‐ If the user successfully logs in following a login rejection, the timer expires as indicated by “…LOGIN_LOCKOUT probation interval for [login] ends.” ‐ If there are 4 incorrect logins within 10 minutes, that login is locked out, indicated by “…login for [login] – failed – user locked out” in the log. To change these parameters, usethe information in Table 65: userlock command on page 344. ‐ “…failed password check” indicates that the user entered the wrong password. ● Login account is indicated in brackets, for example “[craft].” • System originating the request. Sample log: failed Secure Shell SAT login
=
20041110:113254000:2215:lxsec:MED:server_name /usr/bin/sudo: custnsu : TTY=unknown ;
PWD=/opt/ecs/web/cgi-bin ; USER=root ; COMMAND=/opt/ecs/bin/logc -r -c lxsec today
20041110:113232000:2214:lxsec:MED:server_name PAM_unix_auth[3691]: Login for [custnsu] failed - passwd check
20041110:113232000:2213:lxsec:MED:server_name LOGIN_LOCKOUT[3691]: probation interval for
login
[custnsu] begins
20041110:113230000:2212:lxsec:MED:server_name PAM_unix_auth[3691]: Login for [custnsu] from [(null)@services-laptop],tty[NODEVssh]
20041110:112621000:2211:lxsec:MED:server_name logmanager: SAT_auth:Logoff for Sid
[0x800e42d]
20041110:112540000:2210:lxsec:MED:server_name logmanager: SAT_auth:Login for [custnsu] Sid
[0x800e42d] successful
20041110:112540000:2209:lxsec:MED:server_name logmanager: SAT_auth:Login attempt for
[custnsu]
20041110:112538000:2208:lxsec:MED:server_name /usr/bin/sudo: custnsu : TTY=pts/3 ;
PWD=/var/home/defty ; USER=root ; COMMAND=/opt/ecs/bin/sat -A
20041110:112538000:2207:lxsec:MED:server_name PAM_unix_auth[1426]: secure sat connection
detected, changing shell to /opt/ecs/bin/autosat
20041110:112538000:2206:lxsec:MED:server_name sshd[1426]: Accepted keyboard-interactive for
custnsu from 192.11.13.5 port 1265 ssh BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
18
What to look for in this log – ● Login entries without “successful” are attempts only; use the Match Pattern utility at the bottom of the page to search on “failed.” ● Entries containing “root” or “sroot” indicate activity at the Linux root level. Ensure that root access is closely monitored: 20041109:114051000:4270:lxsys:MED:server_name PAM_unix_auth[22971]: Login for [sroot] – successful ● Tripwire changes appear as “doenabletrip,” indicating that changes were made to the Tripwire page. Tripwire monitors changes to files that are expected to change, however Communication Manager purposely does not monitor files that routinely change. ● ASG only: question any login from an IP address other than that for the ASG Guard: 20041109:113504000:4255:lxsys:MED:server_name PAM_unix_auth[21826]: Login for [ION] – from [(null)@161.127.228.32], tty[NODEVssh] Other considerations ‐ ● SNMP trap cannot be set to monitor login/security violations. Changing the lockout parameters – Use the userlock command to change the login probation interval and login attempts. This command is issued at the shell only, not the Maintenance Web Interface. Set up shell access by either: ● Telnet or SSH to the standard ports, respectively, and log in. ● At the Communication Manager SAT type go shell (must have shell access permissions) and press Enter. The command parameters are listed in the table below. Command userlock Argument ‐u login Unlock a locked‐out login
login becomes locked out (use “inf” for infinite attempts). not clear failed login attempts).
permanently lock login out).
‐u login Unlock a locked‐out login
Use
‐t tries Sets the number of unsuccessful login attempts before a ‐i interval Minutes before failed login attempts are cleared (“inf” do ‐o lockout Number of minutes that a login is locked out (“inf” to ‐s show Show current parameters and login attempts. ‐t tries Sets the number of unsuccessful login attempts before a BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
19
2.2.6. Linux login/logout/reboot log This log lists: ● Linux logons and logouts ● System reboots 2.2.7. Linux file transfer log This log lists: ● Information about files copied to or retrieved from the system, including the time, user, and the filenames involved. 2.2.8. Watchdog logs This log lists: ● Application starts/restarts/failures ● Shutdowns and Linux reboots ● Processor occupancy (excessive CPU cycles) ● SNMP traps started/stopped ● Memory ● Process sanity Log entries that are system‐affecting are reported as alarms. This log does not contain hacking/intrusion information, except for terminating an application. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
20
2.2.9. Platform command history log This log lists commands that modify the server administration or status, including software updates that have been installed. Note: For a log of the shell commands that have been executed, look at the Linux syslog or choose the Platform bash command history view from the System Logs page. Note: Sample platform command history log 20041109:220026000:428:cmds:MED:server_name
20041109:220017000:427:cmds:MED:server_name
20041109:220009000:426:cmds:MED:server_name
20041109:220008000:425:cmds:MED:server_name
ipsi-A01a:
20041109:220008000:424:cmds:MED:server_name
20041109:164809000:423:cmds:MED:server_name
20041109:164756000:422:cmds:MED:server_name
20041109:163019000:421:cmds:MED:server_name
20041109:130604000:420:cmds:MED:server_name
20041109:130536000:419:cmds:MED:server_name
20041109:105826000:418:cmds:MED:server_name
20041109:105526000:417:cmds:MED:server_name
20041109:105411000:416:cmds:MED:server_name
20041109:105137000:415:cmds:MED:server_name
20041109:102934000:414:cmds:MED:server_name
20041109:102934000:413:cmds:MED:server_name
root: filesync trans_lsp
logger: fsy_logins
root: /opt/ecs/sbin/filesync ipsi
root: hostscfg -a -I198.152.254.1 -H
root: hostscfg -d -H ipsi-A01a
logger: ip_fw -w
logger: ip_fw -w -q
logger: ip_fw -w -q
logger: ip_fw -w
logger: ip_fw -w -q
craft: productid
logger: ip_fw -w
logger: ip_fw -w -q
craft: /etc/init.d/iptables status
craft: update_show.
logger: swversion
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
21
2.2.10. HTTP/web server error log This log lists errors and events that are generated by the platform Web server, including: ● Web server restarts ● Abnormal CGI script file terminations ● Certificate mismatches This log contains more detail (including IP addresses of the server as shown in the log below) on activity run from the Web interface (including errors) than the Linux access security log. Also,this log shows all actions taken from the Web interface by listing the programs that are run and their parameters. The program names are the key to understanding the action performed. Sample HTTP/web error log 20041109:105526000:2440:httperr:MED:[error] [client 192.11.13.5] w_dolansec running command:
/usr/bin/sudo /opt/ecs/sbin/ip_fw -w 2&gt;&amp;1 , referer: https://192.11.13.6/cgi-bin/
cgi_main?w_lan_sec
20041109:105526000:2439:httperr:MED:[error] [client 192.11.13.5] w_dolansec: calling exec:
sudo /opt/ecs/web/cgi-bin/w_dolansec2, referer: https://192.11.13.6/cgi-bin/
cgi_main?w_lan_sec
20041109:105526000:2438:httperr:MED:[error] [client 192.11.13.5] cgi_main: calling exec :
/opt/ecs/web/cgi-bin/w_dolansec, referer: https://192.11.13.6/cgi-bin/cgi_main?w_lan_sec
20041109:105412000:2437:httperr:MED:[error] [client 192.11.13.5] , referer:
https://192.11.13.6/cgi-bin/cgi_main?susers_menu
20041109:105412000:2436:httperr:MED:[error] [client 192.11.13.5] w_lan_sec running command:
/usr/bin/sudo /opt/ecs/sbin/ip_fw -w -q 2&gt;&amp;1 , referer: https://192.11.13.6/
cgi-bin/cgi_main?susers_menu
20041109:105412000:2435:httperr:MED:[error] [client 192.11.13.5] w_lan_sec: calling exec:
sudo /opt/ecs/web/cgi-bin/w_lan_sec2, referer: https://192.11.13.6/cgi-bin/cgi_main?susers_menu
What to look for in this log ‐ ● “…w_lan_sec2” indicates access to the firewall page; “…w_dolansec2” indicates a change to the firewall settings: 20041109:105526000:2440:httperr:MED:[error] [client 192.11.13.5] w_dolansec running command: /usr/bin/sudo/opt/ecs/sbin/ip_fw ‐w 2&gt;&amp;1 , referer: https://192.11.13.6/cgi‐bin/ cgi_main?w_lan_sec 20041109:105412000:2435:httperr:MED:[error] [client 192.11.13.5] w_lan_sec: calling exec: sudo /opt/ecs/web/cgi‐bin/w_lan_sec2, referer: https://192.11.13.6/cgi‐bin/cgi_main?susers_menu ● Changes to the system configuration appear in the log as “w_config…” 2.2.11. HTTP/web SSL request log BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
22
This log lists all requests made of the Web server’s SSL module, indicating all requested pages or those placed in secure mode. 2.2.12. Communication Manager Restart log This log parallels the display initcauses SAT report that shows the active & standby serveractivity and lists the: ● Last sixteen (16) Communication Manager restarts ● Reason for the request BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
23
2.3.
Select a View This section of the System Logs page allows the user to select a viewpoint for the data in the various logs. Selecting multiple Views might give odd results. The following views are available: ● IP events ● Platform bash command history log ● Communication Manager’s raw Message Sequence Trace (MST) ● Communication Manager’s processed Message Tracer (MDF) ● Communication Manager’s interpreted Message Tracer (MTA) ● Communication Manager’s hardware error and alarm events ● Communication Manager’s software events 2.3.1. IP events This log lists: ● Interfaces (CLAN, MEDPRO, VAL, IP stations) up or down ● Registering/unregistering gateways and IP endpoints ● Reason for IP phone unregistration ● IP address of station registering ● CLAN through which the registration occurred ● Automatic traceroute events To force IP events to appear in this log only, configure the list history command through the SAT interface to omit IP events. The list history log can hold up to 1,500 entries, and configurations with IP endpoints can fill this log quickly. Another alternativeis to send all IP event information to the IP events log, which has a greater capacity than the list history log. Sample IP event log
20041109:131342365:34112:capro(20388):MED:[ IPEVT IPT_REG board=01A07 ip= 161.127.228.23
net_reg= 1 ext= 24100 ip= 161.127.228.47: 3000 net_reg= 1 reason=normal]
20041109:130224805:34083:capro(20388):MED:[ IPEVT IPT_REG board=01B07 ip= 161.127.228.21
net_reg= 1 ext= 24101 ip= 161.127.228.27:49300 net_reg= 1 reason=normal]
20041109:112805025:33726:capro(20388):MED:[ IPEVT IPT_UNREG board=01A07 ip= 161.127.228.23
net_reg= 1 ext= 24101 ip= 161.127.228.27:49300 net_reg= 1 reason=2010
The final entry in the above log lists a reason code of 2010, which exactly matches the Denial Event entry that is logged in the Communication Manager denial event log (display events and type denial in the Category field of the Event Report form). BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
24
2.3.2. Platform bash command history log This log lists all commands that have been issued from the server’s command line interface for the last month. Some acronyms that appear in this log are: ● PPID = parent process ID ● PID = process ID of shell ● UID = is a number that the system associates with a login, for example, “0” is root; all other numbers match to login names. Sample bash history log 20041109:165606000:4426:lxsys:MED:server_name
20041109:164626000:4420:lxsys:MED:server_name
sar01
20041109:164623000:4419:lxsys:MED:server_name
sar01
20041109:164616000:4418:lxsys:MED:server_name
20041109:164613000:4417:lxsys:MED:server_name
20041109:164603000:4416:lxsys:MED:server_name
sa01
20041109:164549000:4415:lxsys:MED:server_name
20041109:164548000:4414:lxsys:MED:server_name
bash: HISTORY: PPID=3266 PID=539 UID=778
bash: HISTORY: PPID=3266 PID=539 UID=778 more
bash: HISTORY: PPID=3266 PID=539 UID=778 file
bash: HISTORY: PPID=3266 PID=539 UID=778 man sal
bash: HISTORY: PPID=3266 PID=539 UID=778 man sa
bash: HISTORY: PPID=3266 PID=539 UID=778 file
bash: HISTORY: PPID=3266 PID=539 UID=778 ls –l
bash: HISTORY: PPID=3266 PID=539 UID=778 ls
2.3.3. Communication Manager’s raw Message Sequence Trace (MST) For use by Avaya technical service representatives. 2.3.4. Communication Manager’s processed Message Tracer (MDF) For use by Avaya technical service representatives. 2.3.5. Communication Manager’s interpreted Message Tracer (MTA) For use by Avaya technical service representatives. 2.3.6. Communication Manager’s hardware error and alarm events BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
25
This log lists the same items that report as display alarms (SAT) or Alarms > CurrentAlarms (Web interface) only in log format. 2.3.7. Communication Manager’s software events For use by Avaya technical service representatives. 2.4.
Select Event Range Use this section of the Diagnostics > System Logs page to refine/restrict the log report: ● Today displays log entries for the current date. ● Yesterday displays log entries for the previous day. ● View entries for this date and time allows the user to specify a date and/or time range. Use any or all of the fields to refine the search. For example, to view the current month’s activity, type the 2‐digit month in the MM field (1st field) only. To view entries for the last hour, type the 2‐digit hour in the HH (2nd field). ● Match Pattern allows users to search for log entries containing the search string that is typed into this field. 2.5.
Display Format ● Number of Lines restricts the log report to a specified number of entries (1‐100,000) ● Newest First lists the most recent log entries first. ● Remove Header removes the sequence number, the log process, and the priority fields from the log entries. This reduces the line length of each entry for easier viewing. Click on the View Log button to view the log report. 2.6.
BTJ
AR RS
Interpreting the log entries Avaya Syslog Implementation Guide
TRS
26
Each line of the log consists of common information, followed by log‐specific information. The beginning of each line contains information, separated by colons (:), and looks similar to the following: 20030227:000411863:46766:MAP(11111):MED: Selecting the merged log view, allows the user to tell from which log the entry was written by looking at the log‐name field on the entry. This field follows the sequence number field, immediately after the timestamp, and is separated by colons. Interpret the information as follows: ● 20030227 is the date (February 27, 2003) ● 000411863 is the time (00 hours, 04 minutes, 11 seconds, 863 milliseconds (ms) or 00:04:11 AM). ● 46766 is the sequence number of this entry. ● MAP(11111), an example from a Logmanager debug trace is the name and number of the process generating the event. Other logs display as an abbreviated name, for example "lxsys" for the Linux syslog and "httperr" for the HTTP/web server error log. ● MED is the priority level (medium). Following the generic information the log‐specific information appears in brackets [ ]. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
27
2.7.
Advanced CM 3.0 Server Log Management Most Linux based servers accomplish their logging through a daemon called syslog. This syslog process writes data from various processes to one of several different log files and to external syslog servers. The syslogd is great for file servers or even web servers, but not for a real‐time call server. This is because a real‐time call server may be trying to log information from various sources simultaneously and the syslog service alone would not be able to handle all of those requests quickly enough. Because of syslogd’s inherent limitations, Avaya CM Linux‐based servers utilize three separate processes capable of writing log files to disk. nbsyslogd is an Avaya developed service that provides non‐blocking access to the Linux syslog daemon. This service provides logging for Linux‐based system and daemon messages, platform commands and security messages. logmanager is a very fast logging service also developed by Avaya. It is capable of logging platform and CM debug information. Such information includes CM restarts, initializations, interchanges, system alarms, IP‐endpoint events, software errors, IPSI messages and packet‐bus information. watchd was developed to monitor process status and provide snapshot CPU utilization reports. It also logs significant events affecting the Linux platform and CM applications. This process actually writes heartbeat information and snapshot reports to its own set of log files on the disk, while at the same time feeding relevant information to logmanager for recording in the debug logs. Log File Rollover CM Linux platforms roll over log files, by default, every 1MB or when a reboot occurs. 2.8.
Configuring syslog Via Linux CLI While normally it will not be necessary to change the configuration of syslog, there is the possibility that it will need to be verified. In addition, beginning in CM 4.0, the ability of the customer to configure an external syslog server is supported. This being the case, it will be essential that a services engineer understand how syslog is configured. The nbsyslogd feeds information into the Linux syslog daemon, as described in the previous section. The syslogd then writes this information to disk into one of several different log files or to an external syslog server. It is possible to configure syslogd via the file /etc/syslog.conf. This file contains a set of rules, which define where different types of events are logged. Each rule consists of three fields: facility; priority; action. These fields are defined in the table below. In the /etc/syslog.conf file these three fields are laid out as follows: facility.priority action BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
28
Facility – identifies the subsystem that generated the log entry used and is one of the following: Auth priv Cron daemon
kern
lpr Mail mark News syslog
use
uucp Local0 Local1 Local2
Local3
Local4
Local5 Local6 Priority – defines the severity of the log entry to be written: Debug info notice warning
err
crit
alert
emerg Action – specifies the destination log file or server for the log entry. Destinations might include: @<syslogserver> /var/log/<logfile> Here is an example of how these fields are used in an actual configuration file. The following is what the syslog.conf file looks like in most pre‐4.0 ACM servers. kern.* *.info;mail.none;news.none;authpriv.none;cron.none;local0.none authpriv.*;auth.* mail.* *.emerg cron.* uucp,news.crit local7.* *.* local0.* local2.* /var/log/kernmsgs /var/log/messages /var/log/secure /var/log/maillog * /var/log/cron /var/log/spooler /var/log/boot.log |/dev/evntpipe /var/log/ecs/commandhistory /var/log/ecs/update.log BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
29
In this file there are several different facility’s being logged to several different log files. Notice that /var/log/messages receives a lot of different input from various sources such as, authpriv, cron, local0. Basically what this shows is that any process which writes log entries using the authpriv, cron, local0, news or mail pipe is going to have its entry written into /var/log/messages. Additionally any type of message that is defined as an info priority will also be written into /var/log/messages. The key is to understand that any process which writes to a log file through the syslogd will define itself as a facility and priority. So in the case of platform commands such as stop –acf, just by looking at the config file above, it is obvious that this command is logged via local0 and written to the file /var/log/ecs/commandhistory. Take note that local0 is also logged to /var/log/messages. So interestingly, this command stop –acf that was just run will show up in both log files. For the most part, the priority field is not important for this discussion. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
30
2.9.
Viewing and Filtering Log Files As previously discussed, the logging capabilities of CM are extremely useful to services engineers and technicians. Virtually anything that happens on a server at any given time is or can be, logged. This allows an engineer to determine the cause of an outage, track intermittent problems or simply analyze performance data. Since so much information can be logged, it can quickly become a daunting task to extract applicable data from these various logs. Next, some of the basic, but useful methods used to view, search and filter log files will be covered. 2.9.1. The logv, logc, logw Commands These three commands are run from a Linux shell and were developed to assist services personnel in quickly searching and filtering the plethora of log file data that exists. The logc and logw commands are both based on logv with a several different options turned on. Table 1 highlights the differences between the three. Most of this information is based on the Linux man page for logv. logv Merge and edit (vi) the various log files in the system logc Merge and output (cat) the various log files in the system logw Watch the requested log file(s)
Table 1: log Commands The best thing about these scripts is their ability to merge and output a range of log files, then to filter that output based on some specified criteria. That criterion might be a date range, a specific event or in many cases a combination of the two. CAUTION!!! The drawback to these commands is that should an engineer type only the command and press ‘enter’ without specifying the type of log that he/she wants to search and a date range, the script will still run. However, it will merge and output every log file that exists in the /var/log/ecs directory. This can cause CPU utilization problems on a server that may already be unstable. It could also drop any dial‐up connections. Most importantly, it will be almost useless to the engineer trying to troubleshoot an issue or determine the root cause of a problem. To see this issue first hand, log into an idle lab switch, go to a Linux shell and type: logc then press ‘enter’. The resulting output is a merge and cat of every *.log file in the /var/log/ecs directory!! How then, could the displayed data be limited? BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
31
2.9.2. Filtering and Searching With logv First, let’s look at some of the most common and useful options for the logv command. The command‐line syntax for logv: logv [OPTIONS] ... [LOGS] ... [‐t TIME] ... [[‐i] FILTERS] [OPTIONS] Most common options ‐l [used alone this automatically pulls up the latest log file in /var/log/ecs] ‐lf <file> [only search specific file, ex. /var/log/ecs/2006‐0731‐034415.log] ‐ld <directory> [search all log files in the specified directory] [LOGS] The type of LOG files to be searched. One or more of the following, see section 4 for more details on the contents of these log files. Table 2: logv Log File Definitions all lm lxboot lxcron lxsys lxsec lxwtmp lxxfer wd cmds httperr Httpssl cmrestart all listed below ‐ don't...just don't
Log manager debug trace (default) ‐ /var/log/ecs*.log
Linux boot messages ‐ /var/log/boot.log
Linux scheduled task log (CRON). /var/log/cron*
Linux syslog ‐ /var/log/messages*
Linux access security log ‐ /var/log/secure*
Linux login/logout/reboot log ‐ /var/log/wtmp*
Linux file transfer log ‐ /var/log/vsftpd.log
watchdog logs ‐ /var/logecs/wdlog*
platform command history log ‐ /var/log/ecs/commandhistory* HTTP/web server error log ‐ /var/log/httpd/error_log*
HTTP/web SSL request log ‐ /var/log/httpd/ssl_requests* Communication Manager restart log ‐ /var/log/defty/initcause.log [‐t TIME] Filter events based on a specific date/time or date/time range. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
32
[yyyy[mm[dd]][:HH[MM[SS[mmm]]]]]][‐[yyyy[mm[dd]][:HH[MM[SS[mmm]]]]]]] [‐i FILTERS] The ‐i option allows the user to specify one or more grep patterns to filter the log by. This option uses an OR operator when more than one entry is specified. NOTE: The FILTER specified for the –i option is NOT case sensitive. This means, hmm and HMM will return the same results for any log entry containing HMM, no matter its case. logv ‐l ‐i hmm Output any line containing an entry for the hmm process logv ‐l ‐i mtcevt Output any line containing a MTCEVT entry logv ‐l ‐i mtcevt hmm Output any line containing MTCEEVT OR hmm For Example: logv ‐lf /var/log/ecs/2006‐0731‐034415.log ‐t 20060801:00‐20060801:2230 This would output all events in the log file specified that occurred between midnight (00:00) and 10:30pm on 8/1/2006. logv ‐ld /var/log lxsys ‐t20060801 This command searches all of the /var/log/messages* log‐files and only displays the entries that occurred on 8/1/2006. Remember, logv merges and outputs log files directly into a vi buffer. This enables the engineer to save the output to a file, use vi search commands or edit the output. 2.9.3. Using logc and logv Both of these commands, logc and logw, are based on logv. In fact, they are simply logv with different options turned on. 2.9.4. Using logc logc is the most similar to logv, except for one minor difference. Instead of sending its output to a vi buffer, logc sends all of its output to a cat stream. logc has the same command‐line syntax and options as logv. One advantage of using logc is that its output can be directed to a text file on the server or even on a different server, without having to use vi. For example: logc ‐l ‐i hmm > /var/home/defty/thisisatextfile.txt BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
33
2.9.5. Using logw logw is perhaps the most useful of these two variations. It makes use of the Linux tail command, allowing the engineer to watch or monitor what is being recorded into the log files. With the exception of [‐t TIME], most of the filters and options used with logv that help to filter its output, can also be used with logw. For example: logw ‐l ‐i hmm mdm mtcevt This would monitor the latest ECS log for entries containing hmm, mdm or MTCEVT. Its output can be watched for as long as necessary and then cancelled with <CTRL>+C. logw is extremely useful for watching a specific type of log file for new entries containing specific data, as in the example above. Though, it may be more useful if the data were captured to a file at the same time. For example: inads@Ozzy> logw ‐l ‐i hmm mdm mtcevt > ~/thisisatextfile.txt & The above command will again watch the ECS logs for output containing hmm, mdm or MTCEVT. Any output that it collects will be directed to a text file in defty’s home directory. The ampersand (&) at the end of this line, allows the command to be run in the background. Engineers can then continue searching other files or perform testing while the log is watched. However, this command should be canceled when finished or before the engineer disconnects from the server. The jobs Linux command will show the running background processes. It simply verifies that the command entered is still active. inads@Ozzy> jobs [1]+ Running logw ‐l ‐i hmm mdm mtcevt >/var/home/defty/thisisatextfile.txt & Once the log data has been collected, in order to cancel the logw command, it must be brought to the foreground using the following syntax. The %1 indicates that this was background job number one. inads@Ozzy> fg %1 logw ‐l ‐i hmm mdm mtcevt >/var/home/defty/thisisatextfile.txt Then a <CTRL>+C will kill the command, returning the user to the shell command‐line. All of the output was saved to the text‐file specified. This can be viewed with vi, copied to another server and eventually printed if necessary. <CTRL>+C inads@Ozzy> vi /var/home/defty/thisisatextfile.txt BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
34
At this point the engineer will be able to use whichever text editor he or she is comfortable with to view and further filter the log data captured. The next section will discuss how to download this data to a computer, the toolsa, and drces systems. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
35
2.10.
Downloading Log Data (Internal) In order to make use of this information, the technician must first know which log or captured data they want to download. See the previous section for more information on dumping log data to a text file. Throughout this section, it is assumed that the log data for download has been isolated and all of this data has been put into a single file. (Internal) For Avaya GSD Engineers, there are four basic steps to take for the most time efficient download of log files from a server. 1. Filter and concatenate log data into a single file, using MTA filter if needed. 2. GZIP Log data into the /var/home/ftp/pub on the server 3. Determine RAS IP address 4. Run ‘scp’ from toolsa or drces to copy the file from remote server to a user’s home directory Step one mentioned above was discussed in the previous section, Filtering and Viewing Logs. Below are step‐by‐step instructions, showing how to download a concatenated log file (/var/home/defty/thisisalogfile.txt) stored on a remote S8500 server that the technician is currently connected to. It is best if the log file is first compressed, especially if the connection to the server is over an analog modem. For the greatest amount of compression, use gzip with the ‐9 option. This tool can turn a 10MB log file into a 500KB download, saving everyone involved a lot of time. To compress the log data, from the server command‐line: init@Amazon‐HQ‐1> gzip ‐9 /var/home/defty/thisisalogfile.txt The previous command will compress the log file and create a file called: /var/home/defty/thisisalogfile.txt.gz Now that the file is much smaller file, the next step is to download it. This is accomplished using the scp command from the toolsa or drces shell. First determine the RAS IP address for the connection. Then scp can to copy a file from the remote server, to the engineer’s home directory on toolsa or drces. For Example: st3tds03 [1012]‐> portall | grep russells 2201 ‐ ‐ ‐ 135.122.53.42 i russells If connected to multiple IP switches, likely multiple lines output will appear with the previous command. Use the dial‐up number, SEID or RAS IP fields in Maestro to determine which connection to copy from. Now, include that IP address in the following command string to copy the log‐file “:/var/home/defty/thisisatextfile.txt.gz” from the remote server, to a file named “~/logdata.txt.gz” in toolsa or drces. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
36
st3tds03‐> scp inads@135.122.53.42:/var/home/defty/thisisatextfile.txt.gz ~/logdata.txt.gz NOTE: At this point the user will be prompted for either a password or an ASG response. The login to use for the ASG challenge would be inads as specified in the above command, though init or craft could also be used. For more details on using scp and the syntax demonstrated above, users can type “man scp“ from the Linux CLI. After a short period of time, long enough for the download to complete, a file named logdata.txt.gz should appear in the user’s directory on toolsa or drces. The final step is to uncompress the log data collected by issuing the command: toolsa: /opt/local/bin/gunzip logdata.txt.gz drces: /usr/add‐on/exptools/bin/gunzip logdata.txt.gz NOTE: If the directory /opt/local/bin (toolsa) or /usr/add‐on/exptools/bin (dr) is in user’s profile’s PATH, the user does NOT need to type the whole path to gunzip. At this point, the download and unzipping of the log data is complete. It is ready for viewing in vi or any text editor. With some practice, this whole process will only take a few minutes or less. If problems are experienced downloading this file, ensure that the correct file name and path are typed. It is also a good idea to verify that the RAS IP address used to download the file from can be pinged. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
37
2.11.
Log File Details (Internal) This section will describe specific details related to each log file of import for services engineers. The details will include file locations and names, required viewing permissions, and a discussion of the most useful contents. Do not consider this to be a complete listing of every important entry at any one time. In an attempt to maintain the accuracy of this information, it will likely be updated often. Be aware too, that likely only two or three of these log files will be useful for a specific case. The information provided herein is intended to be a reference for important log files on the server, allowing the reader to pick which files are useful for any particular scenario. Some of the log files mentioned in this document are not plain text. These special logs are tagged with double asterisks (**). In order to easily view them, it is recommended that logv be used instead of simply vi. If questions arise about how to search for the entries mentioned in this section, see the section entitled “Viewing and Filtering Log Files”. logmanager Debug Trace (AKA ECS) Logs logv ID lm Linux Log File(s) /var/log/ecs/*.log Required Permissions Any login with shell access, read‐only The ECS logs contain debug level information sent to logmanager and syslogd from various CM related processes. This debug information includes CM specific hardware and software errors, duplication link entries, and IP Telephony events such as registrations and even captured MST data. 2.12.
Useful ECS Log Content (Internal) Processes Several significant processes, log errors and miscellaneous information to the ECS logs, these are listed below in Table 3. prc_mgr manages start up of all processes & recovery actions when other processes fail fastmap maintenance action process (map) that runs high‐level maintenance actions like IPSI interchange arbiter makes decision on which server is healthier (input from: pcd, prc_mgr, gmm watchd)
ndm duplication/memory shadowing management
capro call processing including user manager, service dispatcher and conn_m hmm high level maintenance manager prints proc‐errors & maintenance action processes (maps) mdm maintenance data manager
pcd interface between CM & IPSI’s (packet‐interface)
watchd maintains & monitors heartbeats for all CM & platform processes
border communication path between CM and platform processes, i.e., between pcd & arbiter
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
38
2.12.1. Significant Entries (Internal) Each of these entries can be searched for using logv and the filtering techniques described in section 3. NOTE: Searches using any of the log(x) commands ARE case‐sensitive, unless the ‐i option is specified. Segmentation fault – This log entry means that a process has tried to write to or read from, a location in memory to which it was denied access. If this fault is associated with Communication Manager processes, likely CM will have crashed. This requires someone to analyze the Linux core dump and CM mini‐core‐dump files (*.mcd), which were generated along with the crash, in order to isolate the cause. Interchange– Log entries relating to server and IPSI interchanges are written to ECS logs. They include messages indicating the direction of the interchange (ie, which server initiated it) and the reason for the interchange. 2.12.2. Miscellaneous Contents MST Data ‐ Message Sequence Traces if logged, will also be contained in the ECS log files. proc_err – Prcesses errors, also seen via SAT by issuing the display software command. In order to research the meaning of each, the user will need to use the drces system, cscope and be adept at reading code. MTCEVT ERR or ALM – Maintenance Event Errors or Alarms are the same thing seen when running the display errors or display alarms commands in SAT. Normally, SAT commands are the easier way to look at the error log. When viewed inside the ECS logs, errors must be converted from hex data. Type = Hex Error Type Lname = hex MO Pname = hex board location Aux = hex aux data BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
39
2.12.3. Linux syslogd Logs logv ID lxsys Linux Log File(s)
/var/log/messages*
Required Permissions Only sroot can view from the shell Most Linux platform processes will log debugging information to these files, via the syslog daemon. They could be considered a general platform error log, similar to the ECS logs but specifically for Linux platform processes. Probably one of their most useful aspects is that they log contain the history of all commands run from the Linux shell. Also, they contain debugging information for processes such as dhcp and filesync. This information can be very useful when troubleshooting host processes which CM relies on for translation synchronization and backups. The engineer can use this log in conjunction with the ECS logs when performing an RCA. Once the process ID in the ECS log that generated the reset are identified, the engineer can refer to the messages log to see if there are any entries pertaining to that PID and whether user commands could’ve caused the issue. An example of user commands logged to /var/log/messages*: Jan 8 06:12:13 Test‐Server ‐bash: HISTORY: PPID=22396 PID=22397 UID=778 su – sroot Jan 8 06:12:20 Test‐Server ‐2 ‐bash: HISTORY: PPID=22449 PID=22451 UID=0 environment In this example, UID 778 (the init login) executes su – sroot, then as UID 0 (sroot) runs the environment command. This next example we see that an FTP backup has completed to 172.31.5.1. We also know which file‐
sets are being backed up. Jan 8 03:10:02 Test‐Server ‐2 backup: 0: BACKUP SUCCESSFUL for xln backup set, FTP to 172.31.5.1. Jan 8 03:10:02 Test‐Server ‐2 backup: backup image ‐ /Amazon‐HQ‐R13/xln_Amazon‐HQ‐
2_031001_20070108.tar.gz Jan 8 03:10:03 Test‐Server ‐2 backup: 0: BACKUP SUCCESSFUL for os backup set, FTP to 172.31.5.1. If there were errors during this backup, these logs will contain detailed information about the failure. The contents of this log are invaluable when troubleshooting Linux platform processes of any kind. 2.12.4. Linux Access & Security Logs logv ID lxsec Linux Log File(s)
/var/log/secure
Required Permissions Only sroot can view from the shell Security log contents will include login attempts for the system and any security violation information. Such as, who logged in, when and from where. If necessary the engineer can associate the login information from this log with time‐stamps and actions in the messages logs. Jan 8 08:07:31 Test‐Server ‐2 PAM_unix_auth[27315]: Login for [init] ‐ from [(null)@1.2.3.4] on tty[ssh] BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
40
Jan 8 08:07:33 Test‐Server ‐2 PAM_unix_auth[27315]: Login for [init] ‐ successful Jan 8 08:07:33 Test‐Server 2 sshd[27313]: Accepted keyboard‐interactive/pam for init from ::ffff:1.2.3.4 port 2633 ssh2 Here it becomes obvious that someone has logged into the server as init from IP address 1.2.3.4 at 8:07am on Jan 8th. If it was necessary to see which commands they ran, the next step would be to view the messages log and associate actions and PID with the login time. 2.12.5. Linux Kernel Messages logv ID Linux Log File(s)
Required Permissions lxkern /var/log/kernmsgs*
Any login with shell access, read‐only kernmsgs contains very useful information about system hardware, from initialization to failures. It will also log file‐system recovery attempts. The Linux kernel is the interface between system hardware and application software. The data it logs can be helpful when diagnosing hardware issues remotely. For example, on a software‐duplicated S8720 CM 3.0 system, if the second message below never appears, there is likely a hardware or cabling failure of some kind. Dec 27 09:26:01 Test‐Server ‐1 kernel: tg3: eth0: Link is down. Dec 27 09:26:50 Test‐Server ‐1 kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex. However, in this case everything seems to be working fine since the link came up as 1000/full. The contents of these logs will also include items such as, Linux OS and kernel versions, the amount of system memory (RAM) and CPU specific information. 2.12.6. Watchdog Logs logv ID wd Linux Log File(s)
/var/log/ecs/wdlog*
Required Permissions Any login with shell access, read‐only Watchdog is the process that monitors the heartbeats from all important processes on the system and it is responsible for starting, stopping and restarting them when necessary. When something crashes, the information contained in these logs will provide a good overview of which process died or was restarted, first. On CM Release 2.x it also contains information about CPU utilization statistics at regular intervals. Called a CPU Occupancy Profile, this information is similar to the output of the Linux top command. 2.12.7. Platform Command Logs logv ID cmds BTJ
AR RS
Linux Log File(s)
/var/log/ecs/commandhistor
y* Required Permissions Any login with shell access, read‐only Avaya Syslog Implementation Guide
TRS
41
The commandhistory logs provide an excellent overview of which commands have been run on the system. Depending on the release of CM this information may be less than complete. This is because only significant commands run by users or other processes will be logged here. The messages log, on the other hand contains all commands run by any user or process. However, because the messages log will contain much more extemporaneous data than the commandhistory logs, it may be simpler to start with the commandhistory log when investigating significant user actions. As an example, if a user runs save_trans from the shell they would see the entry for that command and any subsequent commands that script internally runs, such as filesync. There is an inherent issue which often arises when commands are run from the web browser by users. The commandhistory log will contain an entry for the command, but it will state that the user logger initiated it. This is not something to worry about, but simply to be aware of. 2.12.8. HTTP/Web Server Logs logv ID httperr Linux Log File(s)
/var/log/httpd/error_log*
Required Permissions Only sroot can view from the shell The http “error_log” is generated by the Apache web server. It contains information about which web pages are being sent to the user, which scripts are being run and which IP addresses are being served. When trying to determine the commands a user has run from the web interface, this would be the log that should be examined. Each command run from the web interface or page accessed will be recorded as an entry containing the full path for the script, any associated arguments and the client which is being served. [Mon Jan 08 13:14:47 2007] [error] [client 135.9.108.67] w_protocols running command: /usr/bin/sudo /opt/ecs/sbin/xinetdserv ‐w ‐s def‐sat ‐q , referer: https://135.9.162.130/cgi‐bin/cgi_main?susers_menu When the web server is restarted this log will also contain information about the startup process. If there is a failure when starting up the web server, the error_log can be examined for entries pointing to the cause. 2.12.9. Linux cron Logs logv ID lxcron Linux Log File(s)
/var/log/cron*
Required Permissions Any login with shell access, read‐only The Linux cron process is a task scheduler. It allows users to run specific commands, including arguments, at specific times and in defined intervals. The cron process logs each command that it runs when scheduled, to /var/log/cron*. These entries will include the command‐line arguments associated with the scheduled task. For example, scheduled backups are cron jobs and logged similar to the following line: BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
42
20061211:031001000:190:lxcron:MED:Amazon‐HQ‐2 crond[4958]: (root) CMD ( /opt/ecs/sys/webbackup ‐b ‐w ‐d ftp://'anonymous':'anon@avaya.com'@172.31.5.1/Amazon‐HQ‐R13 ‐c ‐‐ xln os security) NOTE: When cron logs the backup as shown above, it will also log the password for the ftp account used. In the above example ‘anon’ is the password for user account ‘anonymous’. When troubleshooting backups, for instance, it is possible to copy the entire command string in the log and paste it to the CLI in an attempt to run the backup manually. The command string is all of the text contained inside the parenthesis. 2.12.10.
ACM Restart Log** logv ID Linux Log File(s)
Required Permissions cmrestart /var/log/defty/initcause.log Any login with shell access, read‐only Using logv to display the cmrestart logs is essentially the same as running the shell command ‘restartcause’ or the SAT command ‘display initcauses’. However, the ‘restartcause’ command actually presents this information in an easier to read format. This log is in a special format and cannot be viewed simply by using vi. If it becomes necessary to look directly at this log and it is impossible to use the shell command ‘restartcause’ for some reason, then using ‘logv cmrestart’ is the preferable method. Using the ‘restartcause’ shell command: RESTART CAUSES for Amazon‐HQ‐2 Cause Action Escalated Mode Time Interchange 4 (RELOAD) no Standby 10/24 15:52 Internal Request 1 (WARM) no Standby 10/24 15:52 Using the ‘logv cmrestart’ shell command: 20071024:155200000: 1:cmrestart:MED:Interchange 4 (RELOAD) no 20071024:155200000: 2:cmrestart:MED:Internal Request 1 (WARM) no As can be plainly seen, the ‘restartcause’ command presents the same information in a format that is much easier to analyze. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
43
3. Updates and Changes in CM 4.0 Logging The underlying architecture of both Linux Platform and Communication Manager related logging remains the same in release 4. Debug logs, such as the ECS log files, contain similar data to those in earlier releases of Communication Manager. However, significant changes have been made to the configuration of the syslog service. The following is a summary of changes made to logging in Communication Manager release 4: • Consolidate all syslog data into one of three log files o security events ‐ /var/log/secure o user initiated activity ‐ /var/log/ecs/commandhistory o everything except commands and security events ‐ /var/log/messages o NOTE: syslog data excludes data logged exclusively by logmanager (ie, ECS debug log files) • Log specific details about changes made via SAT or web interfaces. o Unique identity of person making the change (especially if via CMS or PMS) o The parameter being changed and the value of the parameter. • Allow Administrators to choose which events should be logged • Give Administrators the capability to have syslog data sent to an external syslog server for viewing and archiving purposes. o Accomplished via new “Syslog Server” web‐page These changes will now be discussed in more detail. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
44
3.1.
Syslog File Consolidation In prior releases of Communication Manager, data logged by the syslog service was split up into several different categories and then logged to one of eight different log files. This made finding information more cumbersome than was necessary and would have made the implementation of the new features discussed in this section, more difficult. In an effort to simplify, data logged via syslog has been consolidated into one of three log files. Outlined below is the default CM 4.0 /etc/syslog.conf file: # There are 3 local server "logs" # secure security events # commandhistory user initiated activity # messages all except commands and security events # authpriv.*;auth.*;kern.* /var/log/secure local0.* /var/log/ecs/commandhistory *.*;authpriv.none;auth.none;local0.none /var/log/messages These three log files (secure, commandhistory and messages) now contain data that was previously logged to one of eight files. NOTE: Data logged via the logmanager service is not affected by these changes, that data continues to be recorded in /var/log/ecs/*.log and wdlog* files. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
45
3.2.
Enhancements to Logging Command History Prior to release 4, when an administrator executed a command, the parameter changed and the changed values are not logged. For example, if the administrator executes "change station 1000" and changes the station name, the log does not reflect that the name was changed or the value it was changed to. As part of the CM 4 feature requirements, the following change was proposed: It is required that all administrative changes be logged with the time/date, the unique identity of the person making the change, the parameter being changed, the value that the parameter is changed to, and whether the operation was successful or not. It is desirable, but not expected, to log the old values as well. The result of this requirement is that user SAT activity logged to /var/log/ecs/commandhistory now has the format described below: module‐name[pid]: sat sid uid uname uname2 profile R action object qualifier OR module‐name[pid]: sat sid uid uname uname2 profile R action object qualifier fieldName | | newValue OR module‐name[pid]: sat sid uid uname uname2 profile R action object qualifier fieldName | oldValue | newValue module‐name The name of the software module creating the log entry
pid The Linux process ID of the process creating the log entry.
sat The string "sat". This identifies this as a CM SAT log entry.
sid The parent process ID of the autosat process or the process ID of the TUI process associated with this SAT session when access is via CLAN. Allows easy correlation with other activity the same user may have done via a shell. uid The SAT user’s numeric user ID
uname The SAT user’s login name
uname2 The user name of a user of the application logged into CM or if the application logs in as the user, the name of the application. profile The access profile number assigned to this user.
R One of three letter codes as follows: s = action was successful, f = action failed for non‐security reason, v = action failed due to a security violation. The letter "f" may optionally be followed by a colon (:) and an error code in ASCII. e.g. f:123456 action The command invoked by the user. e.g. add, display, list
object The SAT form being worked on. e.g. station, trunk‐group
qualifier The instance of the form. e.g. the 1000 in display station 1000. fieldName The name of the field as it appears on the SAT form
oldValue The value of the field before the change
newValue The value of the field after the change
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
46
NOTE: Communication Manager SAT activity is logged via the syslog facility local0 with a priority of notice. As can be seen by examining the syslog.conf file, local0 data is sent to /var/log/ecs/commandhistory*. 3.2.1. BASH Command Logging Commands run from Linux BASH are sent to syslog facility local0 and logged in /var/log/ecs/commandhistory. If an external syslog server is configured, both SAT and BASH commands would be sent to the same destination. The format for logging BASH commands remains unchanged at this time. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
47
3.2.2. Web Interface Activity User activity performed via the web interface (See Section 2.2) is logged to syslog facility local0, meaning that it would be sent to the /var/log/ecs/commandhistory files. The user information logged with each action contains, IP address of the user, uid, login‐name and profile. The format of log entries pertaining to user activity from the web interface is as follows: module‐name[pid]: web ip uid uname profile R page‐name OR module‐name[pid]: web ip uid uname profile R page‐name | button | button‐name OR module‐name[pid]: web ip uid uname profile R page‐name | variable‐name | value module‐
The name of the software module creating the log entry
name pid The Linux process ID of the process creating the log entry.
web the text string "web" ip the IP address of the user accessing the server.
uid the user id number for the user establishing the web session
uname the user’s login name for the user establishing the web session
profile the access profile number assigned to this user.
R A letter to indicate the status of this action. s = success, f = failure other than for security reasons, v = failure due to a security violation. The letter "f" may optionally be followed by a colon (:) and an error code in ASCII. e.g. f:123456 page‐name The name of the page being accessed, limited to 35 characters. This should be the page name as it appears on the web menu for items which are selected from the main menu, the link name text or a human understandable abbreviation for link text longer than 35 characters for any pages selected from a link not on a menu, or a created name for pages that are not selected by name. e.g. the page which appears for the add button for snmp traps. button the string "button" to indicate the value will be the button identity. button‐name the button label as it appears on the form
variable the possibly abbreviated name of the text box, radio button selection or check box. The name name should always be abbreviated if the string is longer than 35 characters. If the name contains a dynamic part, the dynamic part shall always be logged. For example, the Backup History page presents radio button selections like sv‐manta2.105232‐
20051006.9449. This string needs to be the variable name, not a generic name such as file‐set‐name. value the value of the named variable after change. For example, the string in the entry box or the word checked or unchecked for radio button and check box lists. As the format of these entries indicates, even clicking a button on the web interface is logged. This gives administrators the ability to examine precisely what users are attempting, whether successful or not. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
48
3.2.3. Logging CMS Activity Most 3rd‐party applications, such as CMS, use a single Communication Manager login to make changes from the SAT interface. In regards to CMS, these changes can include moving agents from one split to another, or changing vectors and VDNs. Prior to CM release 4, when 3rd‐party applications made changes via the SAT interface only the application’s login was presented as the user making the change. Starting in release 4, when CMS makes a change via SAT, it will first set what is termed the “secondary user‐name”. The secondary user‐name will contain the user ID of the person logged into and making changes via CMS. For example: some‐name[1234]: mis joe 16 s chg agent‐login I 65103 This allows administrators to see that user ID joe made a change to agent ID 65103, via the CMS interface. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
49
3.3.
How A Configured /etc/syslog.conf Should Look The Communication Manager Server’s /etc/syslog.conf file will contain several new entries after an external syslog server is configured. These should be verified and connectivity to the external server tested, when there are problems sending logs. A fairly small section has been added to the /etc/syslog.conf file on CM servers whether or not an external log server is configured. This section begins with the entry #<REMOTE_LOGGING_START> and continues through the end of the file. Each line of this file beginning with a ‘#’ is commented out and ignored by the syslog service. However, these lines should NOT be modified because the Web interface uses them to properly configure the file. When an external syslog server is configured, for all four logs, the /etc/syslog.conf file should look something like the following example. Example /etc/syslog.conf on a CM 4.0 Server: #<REMOTE_LOGGING_START> #<logging_option> enable #<host_name> 172.31.5.4 #<dup_sync> enable #<srp_sync> enable #<security_log> enable authpriv.*;auth.*;kern.*;local0.none;local1.none authpriv.*;auth.*;kern.*;local0.none;local1.none @172.31.5.4 #<command_log> enable local0.* local0.* @172.31.5.4 #<ip_event_log> enable local1.* local1.* @172.31.5.4 #<message_log> enable *.*;authpriv.none;auth.none;local0.none;local1.none *.*;authpriv.none;auth.none;local0.none;local1.none @172.31.5.4 #<REMOTE_LOGGING_END> enable Again, lines beginning with ‘#’ have no meaning to the syslog service. Take note of the facilities configured in this file, such as local0.* which sends its output to 172.31.5.4. When syslog sends data to an external server, the facility to which the data belongs is also sent. This way when an external server receives syslog data, it can be configured to store that data in a specific file. It is important to remember that the port used by syslog to send data, UDP port 514, must be open between the CM server (outbound) and the external syslog server (inbound). BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
50
4. Configuring Logging in CM 4.0 4.1.
Configuring Syslogging on CM 4.0 and Later Communication Manager Release 4 introduces the ability for customers to have log data collected via syslog, sent to a syslog server on their network. Configuration of this option is done via the Maintenance Web Interface on the CM server as shown in the figure below. Figure 1.1 NOTE: UDP Port 514 needs to be opened through the network, outbound from the CM server and inbound to the syslog server. NOTE: If a server name is used instead of an IP address, be aware that DNS must be already configured on the CM server. Additionally, the hostname must be valid and resolvable. To test, ping the hostname from the Linux shell. If successful, the hostname is sufficiently mapped to an IP address. The top two options give administrators the ability to quickly synchronize this configuration with duplicate servers and ESS or LSP servers. Once configured, this service can also be enabled or disabled without removing the configuration. Clicking the Submit button on this page will apply the configuration and restart syslog on the server. This action is not service affecting. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
51
4.1.1. Setting the Logging Levels Beginning with Communication Manager Release 4, a new SAT screen has been implemented which allows system administrators to choose whether or not to log certain CM‐related events. This new screen is called Logging‐Levels and is shown below. change logging‐levels Page 1 of 2 Enable Command Logging? y Log Data Values: both When enabled, log commands associated with the following actions: add? y export? y refresh? y busyout? y get? y release? y campon‐busyout? y go? y remove? y cancel? y import? y reset? y change? y list? y save? y clear? y mark? n set? y disable? y monitor? y status? y display? y netstat? y test? y duplicate? y notify? y traceroute? y enable? y ping? y upload? y erase? y recycle? y change logging‐levels Page 2 of 2 LOGGING LEVELS Log All Submission Failures: y Log PMS/AD Transactions: y Log IP Registrations and events: y Log CTA/PSA/TTI Transactions: y All of the values on both pages are administrable and either turn on (‘y’) the logging for the specified action or turn it off (‘n’). The events listed on page 2 have been moved from the system‐parameters features form. It should be noted that an administrator cannot change the logging state of all CM‐related events and actions. The following table lists events which are always logged and not administrable. rp wp go tcm
MO
Cumaudits fw‐log
peakaudits No‐lic Software‐events
Test‐number
Internal‐data Angel
mode MO‐all Options Hardware‐group
Enable
Save translations ‐ Old/new values shall be logged according to administration on the logging levels form. ‐ Commands executed within tcm or the debugger are not logged. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
52
4.1.2. Which Log Data Can Be Sent? The External Syslog Server configuration page shows that four log files can be sent to an external server. The following list summarizes each of these files and their contents. Security log • Same as the local file /var/log/secure • All security related events from CM or Linux • Syslog Facilities: authpriv.*;auth.*;kern.*;local0.none;local1.none Command history • Identical to /var/log/ecs/commandhistory. • A log of ALL user initiated activity. • Syslog Facilities: local0.* IP Events •
•
•
NO LOCAL FILE BY DEFAULT This log contains all IPEVT_ entries and a history of “ip‐u” & “ip‐a”. IPEVT_ entries are no longer found in the debug logs, /var/log/ecs/*.log. Syslog Facilities: local1.* Kernel, boot, etc. • Same as the local file /var/log/messages. • Contains everything else logged via the syslog service, including data logged by Linux platform processes such as crond. • Syslog Facilities: *.*;authpriv.none;auth.none;local0.none;local1.none 4.1.3. How To Configure An External Linux Server To Receive Logs Now that the Communication Manager server has been configured to send log data to a remote server, the receiving server must be configured to collect that data. This discussion will be limited to configuring a Red Hat Linux syslog server, though the concepts presented would apply to any type of syslog server. On the customer’s external Red Hat server the first step is to edit one of two syslog startup files. Either /etc/init.d/syslog or /etc/sysconfig/syslog., depending on the version of Red Hat. Normally, if /etc/sysconfig/syslog exists, it is the only one that needs to be edited. There is no harm in editing both files. Once the file is opened for editing, search for the string “SYSLOGD=”. This is a variable which stores startup options for the syslog daemon on the server. Next, verify that the SYSLOGD variable contains at least a “‐r ‐m0”, which tells syslog to listen for remote log data. When correctly configured, the string should at minimum look like the following example: SYSLOGD=”‐r ‐m0” Be sure to save and close the file after editing this variable. Now the /etc/syslog.conf file on the external server must be edited to store the received log data in appropriate files. Remember that syslog will store log data in a file based on the data’s facility and priority, which are written in the syslog.conf file as facility.priority. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
53
If the CM Server was configured to send all four log files to a remote server, as in the preceding examples, it is recommended to also split the received data into four files on the remote server. For example, the following lines could be added to the remote server /etc/syslog.conf: # Communication Manager 4.0 Log Collection authpriv.*;auth.*;kern.*;local0.none;local1.none /var/log/secure local1.* /var/log/ip_events local0.* /var/log/commandhistory *.*;authpriv.none;auth.none;local0.none;local1.none /var/log/messages Any file could be chosen as the destination or all four lines could be sent to the same file. However, bear in mind that all log entries with these facilities will be logged to the file specified, not only entries received from the CM Server. For example, even data on the local syslog server which is logged via facility local1 would be stored in the /var/log/ip_events file. Log entries from a remote server can easily be identified in log files. Since each log entry contains either the remote server‘s name or its IP address, these entries can be filtered out of any log file. The final step, after saving the /etc/syslog.conf file, is to restart the syslog service on the remote server. To verify that syslog is running and listening on UDP port 514 of the external syslog server: root@sylogServer> netstat ‐anp | grep 514 The output should be something similar to the following: udp 0 0 0.0.0.0:514 0.0.0.0:* 25867/syslogd BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
54
5. Troubleshooting Syslog Once syslogging is configured, users should start seeing messages being written to the syslog server. If no messages are being received there are several troubleshooting steps that can be performed to diagnose the issue(s). The issues can range from the syslog service itself to a firewall blocking the port being used by syslog traffic. 5.1.
Firewall Syslog traffic may be blocked both by the firewall on the Communications Manager server or LSP. This can easily be confirmed by viewing the firewall Web Page. This page is available via the maintenance web interface and can be found under the security heading. The figure below shows the Firewall Web Page from Communications Manager 5.0, the web page from Version 4.0 will be similar. Note that in this case, syslog traffic on port 514 is allowed. It should be noted, that syslog traffic may be configured to use a variety of other ports. This may be done for security reasons and will vary by network. Administrators may simply chose an unused port that is not listed in the RFC, or the customer may be utilizing TCP port 1468 for syslog traffic. Syslog traffic may also be blocked at some point on the network path. This can take place due to an access list or a firewall rule that is blocking the port being used by syslog traffic. It may be necessary to involve the network administrator to confirm that the ports for syslog traffic are open. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
55
Firewall Web Page BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
56
5.2.
Review The syslog.conf File Section 3 of this paper shows what a syslog.conf file should look like once properly configured. As part of the troubleshooting process, the syslog.conf file should be reviewed. Specifically, the network address of the log collector should be verified to ensure a mistake has not been made. Below is an example. Areas for review have been highlighted. Review the addresses to make sure that all entries are configured for the correct log collector and that no typos or ommisions have occurred. root@S8710A‐5‐0> cat syslog.conf # # There are 3 local server "logs" # # Log file What # # secure security events # commandhistory user initiated activity # messages all except commands and security events # authpriv.*;auth.*;kern.* /var/log/secure local0.* /var/log/ecs/commandhistory *.*;authpriv.none;auth.none;local0.none /var/log/messages # Alarms *.* |/dev/evntpipe # # The following are administered entries for an external log host. # # NOTE: Do not change the lines between #<REMOTE_LOGGING_START> and # #<REMOTE_LOGGING_END>. # Those lines are managed using the CM "Syslog Server" web page. # The order of these entries is important. # # There are 4 possible external syslog server logs # # Log file What # # secure security events # command history user initiated activity # IP events IP registrations and events # messages all except commands, security events, and IP events #<REMOTE_LOGGING_START> #<logging_option> enable BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
57
#<host_name> 10.1.1.50 IP address or DNS name of the syslog server. In this example, and in the entries below, 10.1.1.50 is the address of the syslog server. #<dup_sync> enable #<srp_sync> enable #<security_log> enable authpriv.*;auth.*;kern.*;local0.none;local1.none authpriv.*;auth.*;kern.*;local0.none;local1.none @10.1.1.50 #<command_log> enable local0.* local0.* @ 10.1.1.50 #<ip_event_log> enable local1.* local1.* @ 10.1.1.50 #<message_log> enable *.*;authpriv.none;auth.none;local0.none;local1.none *.*;authpriv.none;auth.none;local0.none;local1.none @ 10.1.1.50 #<REMOTE_LOGGING_END> enable ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
58
5.3.
Check the syslogd Service It is possible that the syslog service is not started, or may need to be restarted. There are several methods to check the syslog service. Note, that administrators will need root level access to perform these functions. Start by checking the /var/log/messages file to see if the message files exist and what dates they were created on. If there are no log files then the service may need to be started/restarted. Additionally, the syslog service can be checked in a variety of ways including the two listed below: /sbin/service syslog status ps aux | grep syslogd There are cases that even though the syslog service is running, logging still is not taking place and the service will still need to be restarted. Check the message log for any errors as the service starts to determine if other problems exist. Use the following command to restart the syslog service: /sbin/service syslog restart BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
59
6. Event Logging for 4600 and 9600 Series Telephones The following escalation provides a very good example of how endpoint event logging can be useful. Recently a customer required a firmware upgrade on their 9600 phones. The engineer was unable to upgrade the firmware. The engineer escalated the issue, since it was unclear what the phones were actually doing, and why the firmware upgrade was unsuccessful. Had syslogging been enabled on the phones, the engineer would have had a much clearer picture of what issues were actually taking place. This could have prevented, or at least reduced the escalation time spent on the problem. 6.1.
Configuration Logging set up for 4600 and 9600 series telephones is done through the 46xx settings file. Below is an example of this file. Logging levels, the syslog server IP address, and the facilities are set inside this file. Note that local logging is not supported on the 9600 series phones. Note, that only the two highlighted entries in this file (shown below) can be changed. ################ EVENT LOGGING SETTINGS ################## ## ## Event Logging control ## Controls the level of events logged in the ## endptRecentLog and endptResetLog objects in the SNMP ## MIB. Events with the selected severity level and higher ## will be logged. ## LOGLOCAL is not supported on 96xx SIP phones. ## 0 for disabled ## 1 for emergencies ## 2 for alerts ## 3 for critical ## 4 for errors ## 5 for warnings ## 6 for notices ## 7 for information ## 8 for debug SET LOGLOCAL 5 This entry can sets the logging level from the commented choices above. In this example, the logging level is set to Level 5 for Warnings ## ## Syslog Server address ## One syslog server IP Address in dotted‐decimal or DNS ## name format (0 to 255 ASCII characters). SET LOGSRVR 192.168.0.15 Sets the syslog server that the endpoint will send logs to. ## ## BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
60
6.2.
Enabling/Disabling Logging on Endpoints Event logging can be enabled/disabled at the telephone level. Remember, however, that the actual syslog server and the facilities can only be set inside the 46xx settings file. Use the following procedure to enable or disable logging of system events. Changing the log settings on the endpoint allows an engineer to troubleshoot a specific handset(s), without changing the logging levels on all handsets at a location. 1. Select LOG from the Craft Local Procedure Screen, the telephone prompts to use the Right and Left navigation arrows to select a setting and displays the following text: Where, except for the 9610 which displays the actual value, the text string is the wording associated with the current system value of NVLOGSTAT, defined as: ● "Disabled" when NVLOGSTAT = 0 ● "Emergencies" when NVLOGSTAT = 1 ● "Alerts" when NVLOGSTAT = 2 ● "Critical" when NVLOGSTAT = 3 ● "Errors" when NVLOGSTAT = 4 ● "Warnings" when NVLOGSTAT = 5 ● "Notices" when NVLOGSTAT = 6 ● "Information" when NVLOGSTAT = 7 ● "Debug" when NVLOGSTAT = 8 2. To change the setting, press the Right (or Left) navigation arrow to cycle through the settings. Depending on the current value, the next sequential text string is selected and displayed as the setting. For example, if the current value is Alerts (2), pressing the Right navigation arrow changes the value to Critical (3). If the current value is Debug (8), pressing the Right navigation arrow changes the value to Disabled (0). 3. Press Save to store the new setting and redisplay the Craft Local Procedure screen. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
61
7. Gateway Logging 7.1.
High Level Feature Description Fundamental to end‐to‐end logging is the existence of logs at the device/element level. That is the lowest level application, hardware component, etc. one would want to isolate during fault isolation/debugging. These logs come in two general forms: • Externally consumed logs for various levels of technicians to trouble‐shoot the system. These logs are of default or standard type and are always available such as syslog. • Internally held detailed developer level logs (e.g. software debugs) which often real time systems cannot keep up with the enormous amount of data produced. These detailed logs are not typically forwarded or centralized, although they can be made accessible on demand on the gateway Command Line Interface (CLI) command show event‐log available to those able to access tech mode. The focus of this section is on externally consumed logs such as syslog. The requirements in this section apply to media gateways: G250, G350, G450, IG550, and IG480. The G700 is not covered in this document since it does not have the same underlying architecture and code base. In addition, G700 will soon be replaced by G450. 7.2.
Media Gateway Severities The media gateway processors currently have the following four severities for their SNMP notifications: ƒ
Critical ƒ
Major ƒ
Minor ƒ
Info These media gateway processor messages are all lumped into a single severity classification of “informational” when these messages are incorporated into the kernel syslog messages from the media gateway (G700, G450, G350, G250, TGM550, TRM480) which have eight different severities based on RFC 3164. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
62
As a consequence, all four different severities (Critical, Major, Minor and Info) do not map to the syslog standard RFC 3164. By default, all entries without assignment of severity are logged as informational. Since the media gateway processor severities do not align with standard syslog message severity, a reader of the log can be easily confused about the true severity of the media processor message. This is particularly important when other devices or equipment also send messages to the same syslog server. This is particularly true for gateway platforms that are boards inserted into a chassis manufactured by another company (i.e. TGM550 or TRM480). Table 3 – Current Mapping of MG Severities to RFC 3164 Severity Levels Current MG Severities RFC 3164 Severity Level
Emergency
Alert
Critical
Error
Warning
Notice
Critical, Major, Minor, Info Informational
Debug
Notice Informational Debug Info
5
6
7
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
63
7.3.
Media Gateway Syslog Message Format As described in Section 1.5, the Syslog message has three parts: the PRI (with angle brackets), followed by the HEADER (date, time and hostname or ip address), and the the MSG. The total length of the packet MUST be 1024 bytes or less. However, in the gateway the maximum packet size is 512. There is no minimum length of the Syslog message although sending a Syslog packet with no contents is worthless and SHOULD NOT be transmitted. The outline of Syslog message is shown below. Note that even though the message has a severity of “major” or “error,” the MSG lists the problem description as “VOICE‐Informational.” The correct severity level is not listed (Note: this has been corrected in media gateway releases associated with CM 4.1 – in which severity is properly listed as “VOICE‐Error”. For information on firmware levels and their compatibility with Avaya devices, please see the compatibility matrix at the link below. http://support.avaya.com/elmodocs2/comm_mgr/CM_SW_FW_Compatibility_Matrix.doc <187> Oct 11 22:14:15 TGM550 MSY‐TRPMAJNA‐00038[VOICE‐Informational: Primary Controller not found Date and
PRI
Hostnam
Mnemoni
MSG
HEADER
Figure 2 – RFC 3164 Syslog Message format This current format lacks time stamp precision less than one second. In addition, it does not
contain any unique identifying information regarding the gateway. This document specifies
requirements to pre-append millisecond time stamps to the beginning of the MSG text and
append the Authentication File ID to the end of the MSG text.
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
64
7.4.
Media Gateway Syslog Filtering Currently, syslog messages are sent according to the syslog server based on a fixed hierarchy of severity levels. When one severity is specified as the filtering criteria for syslog messages, it is assumed that all events with that severity level or “higher” in severity will be sent. For example, if the user specifies syslog to report all notifications, then all messages with the severity of Notice (code 5) through emergency (code 0) are sent to the syslog server by default. In some cases, not all of these messages are desired because the volume of traffic this may potential generate on the network. Users may want to specify only one severity level or mix and match different severity levels. The current media gateway implementation of syslog does not allow for this type of filtering. This document specifies requirements modifying an existing CLI command “set logging file condition” is specified to allow for more precise filtering of syslog messages based on a single severity or mix of severities. 7.4.1. Architectural Overview (Internal) In the current implementation, media gateway processor messages originate from the maintenance module (mtce) which are sent to the logger. From the logger, these messages are sent to a development log (see red lines in the figure below). These developer logs may be accessed on the media gateway using the Command Line Interface (CLI) executing the show events‐log command in tech mode. This log is voluminous and verbose, containing a great deal of detail for debugging purposes. In addition, the logger will also send these messages to the media gateway’s kernel. The kernel sends these messages to the SNMP agent (see purple dot‐dash lines in Figure 1 below). The SNMP agent sends SNMP notifications to the CM server’s SNMP receiver by default where they are entered into the server’s syslog. As many as ten other SNMP receivers (such as Network Monitor Console (NMC) or HP Openview) may be specified in the media gateway CLI. These SNMP notifications contain severity levels: critical, major, minor, and info. The media gateway kernel will also internally process these messages from the logger into its own syslog process in the media gateway (see blue dotted lines in Figure 1 below). In this process, the 4 severity levels found in the SNMP notifications (critical, major, minor and info) are lost. The media gateway can send syslog messages to a 3rd party syslog server (as many as 3 servers may be specified as destinations). The severity levels used with this syslog process are Emergency, Alert, Critical, Error, Warning, Notification, Informational, and Debug. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
65
Figure 3. Media Gateway log information flow BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
66
7.4.2. Interpretation of Syslog server entries from a media gateway Here is sample of a syslog entries found in a syslog server from a TGM550 with a firmware release associated with CM 4.1. Apr 04 12:15:15 local Listening for Syslog messages on IP address: 135.8.76.107 Apr 04 12:23:11 192.168.40.39 APR 4 12:20:59 192.168.40.39 MSY‐TRPMINNA‐00032[VOICE‐Warning: H248 link is DOWN Apr 04 12:24:37 192.168.40.39 APR 4 12:22:24 192.168.40.39 MSY‐TRPMAJNA‐00038[VOICE‐Error: Primary Controller not found Apr 04 12:24:37 192.168.40.39 APR 4 12:22:24 192.168.40.39 MSY‐TRPMINNA‐00036[VOICE‐Warning: Registration successful (192.168.40.39, SLS) Apr 04 12:24:37 192.168.40.39 APR 4 12:22:24 192.168.40.39 MSY‐TRPMINNA‐00032[VOICE‐Warning: H248 link is UP Apr 04 12:24:37 192.168.40.39 APR 4 12:22:24 192.168.40.39 MSY‐TRPMINNO‐00047[VOICE‐Warning: Admin resetting angel Apr 04 12:24:37 192.168.40.39 APR 4 12:22:25 192.168.40.39 MSY‐TRPMINNO‐00001[VOICE‐Warning: Media Module went INSANE! Apr 04 12:24:40 192.168.40.39 APR 4 12:22:28 192.168.40.39 SYN‐TRPMAJNO‐00002[VOICE‐Error: Clock sync failure FAULT Apr 04 12:24:48 192.168.40.39 APR 4 12:22:35 192.168.40.39 MSY‐TRPMINNO‐00001[VOICE‐Warning: Media Module insertion successful! Apr 04 12:25:00 192.168.40.39 APR 4 12:22:48 192.168.40.39 SYN‐TRPMAJNO‐00002[VOICE‐Error: Clock sync failure CLEAR Apr 04 12:33:20 192.168.40.39 APR 4 12:31:08 192.168.40.39 CliCommand[CLI‐Notification: root: copy run start Apr 04 12:34:16 192.168.40.39 APR 4 12:32:03 192.168.40.39 CliCommand[CLI‐Notification: root: show ex Apr 04 12:34:21 192.168.40.39 APR 4 12:32:08 192.168.40.39 CliCommand[CLI‐Notification: root: exit Apr 04 12:36:33 192.168.40.39 APR 4 12:34:20 192.168.40.39 CALL COMPLETED[CDR‐Informational: 12:33 00:00 A 115 60011 60209 v102 Apr 04 12:36:33 192.168.40.39 APR 4 12:34:21 192.168.40.39 CALL COMPLETED[CDR‐Informational: 12:33 00:00 A 115 60022 60142 v101 In the sample syslog entries above, the first line declares the date and time the syslog file started logging events. On the second line, the first entry is logged listing the syslog server’s date and time stamp (Apr 04 12:23:11) indicating when it received the syslog message from the media gateway followed by the ip address of the media gateway (192.168.40.39). Next, the HEADER information is presented with the media gateway’s date and time stamp (APR 4 12:20:59) appears followed by the gateway’s ip address (192.168.40.39) since no hostname was specified. Then, the mnemonic appears (MSY‐TRPMINNA‐
00032) with the left square bracket as a delimiter signifying the end of the HEADER and the beginning of the MSG part of the syslog message. The mnemonic lists the subsystem that originated the message (MSY) and the SNMP trap (TRPMINNA). MSY stands for the Maintenance sub‐system (for more definitions see Appendix A). The MSG part begins with “VOICE‐Warning:” which indicates that the message is from the Media Gateway Processor (MGP) followed by a dash and the severity of Warning with a colon “:” and space preceding the explanatory text “H248 link is DOWN.” Then subsequent entries show that the media gateway could not find the Primary Controller (a CM server) in line 3. Next, in line 4, the media gateway enters SLS mode (Standard Local Survivability) as it re‐establishes its h.248 link to the Survivability Call Processing engine inside the media gateway in line 5. This leads the media gateway to reset its media modules leading to events listed in lines 6 thru 10. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
67
After successfully entering SLS mode and resetting media modules, a technician issues CLI commands to save the current configuration file using “copy run start” and listing extensions using “show ex” while logged in as root in lines 11 thru 13. After exiting SLS mode in the CLI in line 13, Call Detail Records (CDRs) capture phone calls that were made in SLS mode while the media gateway is operating without an h.248 link to the CM server. The time of the call is listed (12:33) as well as the duration of less than one minute (00:00 – hours: minutes) using a Trunk Access Code (TAC) of (115) to call extension 60011 from extension 60209 via a trunk provisioned on a media module in slot 1 connected to port 2. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
68
7.4.3. Provisioning and Installation Description The media gateway syslog feature is enabled by using the CLI command “set logging” to specify the syslog server with an ip address as well as setting the severity criteria for sending syslog messages to the server. Here’s an example of the commands used to configure the media gateway for syslog messages to be sent to a syslog server. set logging server 135.8.76.107 set logging server condition all Informational 135.8.76.107 This set of commands can be saved in the start‐up configuration file. 7.4.4. CPU / Memory / Bandwidth / Resource Needs Currently, most of the gateways have 50 KB of flash to save internal logs. CTO standards have increased this requirement to 4 MB. The only gateway to satisfy this requirement is the TRM480. Table 4: Current Memory Usage FLASH MEMORY USAGE1
G250, G350, G450, TGM550 FLASH
50 KB
TRM480 FLASH 4 MB
1
This pertains to the S8300C and S8400 platforms.
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
69
7.5.
Branch Gateway G450 Syslog Configuration Configuration of the G450 is relatively straight forward. Below is a basic command line configuration. A short explanation of each command appears in blue. set logging file condition {all|<application>} {none|<severity>} Shows the conditions that can be set for logging G450‐001(super)# set logging file condition ARP DHCPS OSPF SAA USB BOOT DIALER POE SECURITY USB‐MODEM CDR DNSC POLICY SNMP VJ‐COMP CLI FAN PPP SUPPLY VLAN CNA_TP FILESYS PPPOE SWITCHFABRIC VOICE CONFIG IDS PROXY‐ARP SYSTEM WAN CONSOLE IPHC QOS TFTP all DHCP‐RELAY IPSEC ROUTER THRESHOLD DHCPC ISAKMP RTP‐STAT TRACKER G450‐001(super)# set logging file condition all In this case, the gateway will be set to log all conditions. G450‐001(super)# set logging file condition all {severity} Alert Debug Error Notification none Critical Emergency Informational Warning Lists the severity of the incidents to be logged. none = disables logging messages for the specified application (condition). G450‐001(super)# set logging file condition all debug Done! This command sets the logging level to debug. CAUTION: Everything will be logged if this is set, possibly causing CPU issues with the system. G450‐001(super)# show logging file condition Verify logging ****************************************************** *** Message logging configuration of FILE sink *** Sink Is Enabled Sink default severity: Debug G450‐001(super)# BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
70
G450‐001(develop)# set logging server ? Lists server commands to set the log collector. Set logging server commands: ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ Syntax : set logging server <ip_addr> ip_addr ‐ ip address of the syslog server set logging server access‐level Set logging server access‐level, use 'set logging server access‐level help' for more info set logging server condition Define a filter for sending logging messages to the log file set logging server disable Disable sending logging messages to the syslog server set logging server enable Enable sending logging messages to the syslog server set logging server facility Set server facility parameter for the specified syslog server NOTE: Optionally, define an output facility for the Syslog server by typing the set logging server facility command, followed by the name of the output facility and the IP address of the Syslog server. If the user does not define an output facility, the default local7 facility is used. For example: set logging server facility auth 147.2.3.66 The following is a list of possible facilities: auth. Authorization, daemon. Background system process, clkd Clock daemon, clkd2 Clock daemon, mail Electronic mail local0 – local7 BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
71
7.5.1. G450 Syslog Example G450‐001(super)# set logging server 135.148.73.180 Done! G450‐001(super)# set logging server condition all debug 135.148.73.180 Done! G450‐001(super)# set logging server enable 135.148.73.180 Done! G450‐001(super)# show logging server condition ****************************************************** *** Message logging configuration of SYSLOG sink *** Sink Is Enabled Sink default severity: Debug Server name: 135.148.73.180 Server facility: local7 Server access level: read‐write G450‐001(super)# Example of the G450 Syslog ouput 003/21/2008,12:50:12:ARP‐Debug: Received ARP request: src 135.122.49.195 00:07:3b:90:2a:c8 dst 135.122.49.193 00:00:00:00:00:00 on interface Vlan 60 03/21/2008,12:50:12:ARP‐Informational: Added dynamic ARP entry for IP address: 135.122.49.195 00:07:3b:90:2a:c8 on interface Vlan 60 03/21/2008,12:50:12:ARP‐Debug: Received ARP response: src 135.122.49.195 00:07:3b:90:2a:c8 dst 135.122.49.194 00:04:0d:ea:ab:30 on interface Vlan 60 03/21/2008,12:50:12:ARP‐Debug: Sent ARP request: src 135.122.49.194 00:04:0d:ea:ab:30 dst 135.122.49.195 00:00:00:00:00:00 on interface Vlan 6003/21/2008,12:50:12:ARP‐Debug: Creating unresolved ARP entry for IP address: 135.122.49.195 on interface Vlan 60 03/21/2008,12:50:12:ARP‐Debug: Sent ARP request: src 135.122.49.194 00:04:0d:ea:ab:30 dst 135.122.49.194 00:00:00:00:00:00 on interface Vlan 60 03/21/2008,12:50:12:ARP‐Debug: Added local ARP entry for IP address: 135.122.49.194 00:04:0d:ea:ab:30 on interface Vlan 60 BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
72
03/21/2008,12:50:11:VOICE‐Error: Clear No Call Controller 03/21/2008,12:50:11:VOICE‐Error: No Call Controller Found 03/21/2008,12:48:58:CLI‐Notification: root: reset 03/21/2008,12:48:54:CLI‐Notification: root: set boot bank bank‐a 03/21/2008,12:28:54:FILESYS‐Notification: Firmware Download Success 3/21/2008,12:26:59:CLI‐Notification: root: copy ftp SW_imageA g450_sw_27_27_0.bin 135.148.73.180 03/21/2008,09:23:33:VOICE‐Error: Registration failed (135.122.49.195 U, Primary) 03/21/2008,09:23:28:BOOT‐Informational: Booting from bank B with firmware version 27.26.0 03/21/2008,09:23:28:BOOT‐Informational: System boot up from warm reset 03/21/2008,09:23:26:VOICE‐Error: Clear No Call Controller 03/21/2008,09:23:26:VOICE‐Error: No Call Controller Found 03/21/2008,09:22:13:CLI‐Notification: root: reset 03/21/2008,09:19:00:CLI‐Notification: root: set logging file condition all Debug BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
73
8. TN Board Syslogging (Internal) The following TN boards support syslogging: CLAN (TN799) Crossfire (TN2602) IPSI (TN2312) 8.1.
CM 5.1 and Later System‐wide Settings This CM form sets the system‐wide default settings for syslogging from TN boards. The facility, destination syslog server IP addresses, and ports can be set from this page. Up to three syslog servers can be administered at this single point. This allows administrators to implement some redundancy into the syslog design in the event of network or syslog server failures. For example, a company with two CM servers may log to one main syslog server; however, if that main syslog server is lost, logs will be sent to the secondary or tertiary server if they are defined on this form. The following are system‐wide changes in Communications Manager via SAT: BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
74
8.2.
Syslog on CLAN (TN799) and Crossfire (TN2602) Boards (Internal) From the SAT screen , run the change ip‐interface with the location of the IPSI or Crossfire board. Add new page 2 on the change ip‐interface form for C‐LAN and XFIRE to support administration for Syslogging a. Login on switch with ‘init’ or ‘inads’. b. Enable Ethernet Port; field is set to ‘y’ (page 1) Syslogging from TN boards is limited to “init” and “inads login only. Therefore customers and business partners will not be able to access this feature. The Ethernet port (page 1 of the command listed above), on these TN boards is not enabled by default. In order for the board to have network connectivity to the syslog server, it must be enabled. The object levels on this page allow administrators to specify the processes on the board that need to be debugged on the board. The level field specifies the debug level for a particular object. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
75
8.3.
Syslog on IPSI (TN2312) Boards (Internal) The ipserver‐interface form supports administration for syslogging. Login on switch with ‘init’ or ‘inads’. Note that since the IPSI requires constant communication with the CM server, there is no need (and no option to enable an Ethernet interface). BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
76
8.4.
Enabling Syslog with Duplex Crossfire (TN2602) Boards (Internal) A duplicated Crossfire board (TN2602) has two “Enable Syslog” fields. Both the active and the standby boards can send syslog messages. It is possible to have one board with newer firmware send log messages, while the board with the older firmware can have syslogging turned off. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
77
8.5.
Changing Syslog Features With the TN Board Command Line These features can also be changed at the TN Board command line. This scenario may become necessary if a client is running CM versions prior to 5.0. In that case, syslogging at the system‐
wide level is not supported. Logging must be enabled at the board level. Logging may be set from the TN board command line as follows: Set syslog server <serv #>< enable><facility><severity><AF><IP String><Port #> 1‐3 16 ‐23 0‐7 "ipv4" IP add 0 ‐ 65535 or "ipv6" Enable example: set syslog server 1 enable20,7,"ipv4", "172.22.22.10", 514 Which would set up Receiver # 1 to receive messages with facility local4, severity = debug, “ipv4” ip addressing to 172.22.22.10 at port 514. Disable example: set s yslog serverSet 1 disable The above command line options are consistent across the three boards. To see what the current syslog server settings are on the board for an IPSI, from the command line perform the following: [IPADMIN]: show syslog server f s p e a e o s n c v r e a i e t r b l r v l i i n e e t t u r d y y aF ip_address m ‐ ‐ ‐‐ ‐ ‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐ 1 y 20 7 ipv4 135.9.76.78 514 2 y 20 7 ipv4 135.222.222.222 314 3 y 20 7 ipv4 135.222.222.223 65300 TO set the debug levels and objects: [IPADMIN]: syslogDebugSet <debugObject> <debugLevel> BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
78
8.6.
Logging Into TN Boards 8.6.1. IPSI Board To log directly into an IPSI board and make the changes described in section 8.5, use the following procedure. If the IP address of the board is known, skip steps 1 and 2; simply telnet to the board and perform steps 4 and 5. 1. Using a crossover Ethernet cable, make a connection between the serial port on a laptop/desktop and the services port located on the IPSI board. 2. Configure the following settings in Hyperterm, or other desired terminal emulator a. Bits per second speed 9600 b. Data bits 8 c. Parity None d. Stop bits 1 Off e. Flow control 3. Once these settings are complete, obtain a login prompt at the IPSI board. 4. Type: ipsilogin and enter the proper credentials to access the IPSI board. 5. At this point, utilize the procedures described in Section 8.5 to enable and disable logging features on the board. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
79
8.6.2. Logging into CLAN and Crossfire Boards Before a telnet session to a CLAN or Crossfire board is possible, some simple administration on Communications manager must be performed. 1. From the SAT interface, type: enable session 2. On the SAT form that will be displayed, enter a user name, password, and define the board location that the CLAN or Crossfire boards reside. 3. Once this administration is complete, the user will be able to telnet to the board using the user name and password defined above. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
80
8.7.
IPSI Debug Logging and Levels Today IPSI sends data to CM log files. If Syslogging is enabled for the IPSI without modifying the debug objects and levels, the data that is sent to CM will also be sent to the syslog servers. Debug objects and levels are used to troubleshoot specific problems with the IPSI. In most cases the developer/SSE will tell the user what the debug objects and levels should be when they are collecting the logs for the IPSI. Syslogging pretty much eliminates the need to attach a firmware monitor to the IPSI to gather detailed log events. Debug objects range from 0‐255 and debug levels range from 0‐65535. Setting the object to 255 and level to 65535, it results in an IPSI log dump 8.7.1. IPSI Objects and Levels Object Levels • 0x000 ‐ NONE • 0x001 ‐ Error • 0x002 ‐ Firmware Status • 0x004 ‐ DataFlow • 0x008 ‐ Object Status • 0x010 ‐ Function Flow • 0x020 ‐ Periodic Maintainence • 0x040 – Misc Objects • 0 ‐ AANGEL • 1 ‐ Alarms • 2 ‐ Angel • 3 ‐ Angeld1 • 4 ‐ ANGELSF • 5 ‐ Angelul • 6 ‐ Counter • 7 ‐ Evmon • 8 ‐ Ess • 9 ‐ Facttest BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
81
8.8.
Firmware Levels for TN Board Syslogging Communications Manager 5.1 supports syslogging from TN boards utilizing the following
firmware:
IPSI (TN 2312) Version 44x and later
CLAN (TN799) DP HW 13+ Version FW27
CLAN (TN799) DP HW 0‐10 Version FW28
Crossfire (TN2602) Version FW40
8.9.
Caveats for TN Board Logging •
•
CM SAT settings will override any settings made form the command line on the board •
Business partners and customers will not be able to see this feature because it is restricted to init and inads login. •
If all three destination syslog ip addresses are the same, only one stream of data is sent by the board and SAT will not block such an administration. •
If the boards have a firmware version that does not support syslogging, then syslogging cannot be enabled from SAT. If there are a large number of boards on the CM, and the administrator forgot which ones were enabled, there is no disable all/ipsi/clan/crossfire command currently. It is being implemented and will most likely be in CM5.2 BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
82
9. Enabling Syslog in Sip Enablement Server (SES) Setting up syslogging in an SES server is very similar to the CM configuration.
Log into the SES web interface and choose Syslog Server under the Security heading. The screen
below should appear.
Checking the first box allows the syslog configuration to be written to connectedLSP and ESS
servers.
Check the radio button to enable syslogging. Add the DNS name or the IP address of the server
that will be collecting the syslog messages.
The next series of check boxes allows the administrator to choose what types of events our
logged, security, command history, etc. These boxes can be enabled and disabled on an asneeded basis.
Click the Submit button to submit changes to this page.
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
83
10. Syslog and CMS Currently, Avaya Call Management System (CMS) does not support syslog for application layer logging. However, operating system level logs can be utilized to detect suspicious system activity. The customer may review the following log files on a routine basis for signs of unusual activities: /var/adm/messages ‐ This log will include system messages (syslog), including failed login attempts if auth.debug is enabled in /etc/syslog.conf /var/adm/sulog ‐ This log contains su records for access to root privileges. /var/cron/log ‐ This log contains cron records for automated scripts run under the cron process. 11. Syslog and Intuity Audix Intuity Audix does not currently support syslog. Syslogging may be configured at the OS level, but these operating system level events will have little relation to Intuity Audix. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
84
12. Syslog and Modular Messaging Syslog capabilities for Modular Messaging are currently available in releases 4.x and 5.x.
In these releases basic syslog capabilities can be set up on the MAS only. The MSS box
is a Linux back end that functions as a message storage server. The MSS does not have
any special syslogging capabilities beyond what can beset in the Linux OS.
12.1.
Syslog Configuration Configuration of syslog capabilities is identical in versions 4.x and 5.x.
1. In the Voice Mail System Configuration window, click the voice mail domain. 2. Double‐click Auditing. The system displays the Auditing dialog box for the selected voice mail domain, with the General tab active. 3. Click the Syslog tab, the following screen is displayed BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
85
12.1.1. Events to Send to Syslog Use the “Events to Send to Syslog” setting to filter the events that are sent through syslog. If All events is selected, then all events are sent through syslog. If Custom events from the same dropdown list, then the check boxes are activated. The check boxes can then be enabled to customize the types of events as follows: . Allowed events Enable to send all allowed events through syslog. . Denied events Enable to send all denied events through syslog. . Error events Enable to send all error events through syslog. 12.1.2. Adding the Syslog Server The Syslog Servers box above displays a list of machines that can receive events through syslog. The name of each machine can be an IP address, machine name, or fully qualified domain name. Adds a line to the list. The user can then enter the name of the machine or select it using the browse button (...). Deletes the machine selected in the list. The system prompts the user to confirm the deletion.
Moves the selected machine up or down the list.
Note: The syslog server added here does not need to be a Modular Messaging system. Any log collector capable of receiving syslog traffic will work. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
86
13. Syslog and the Audio Codes Gateway 13.1.
External Syslog Syslogging on the Audio Codes gateway can be configured for internal or external logging. External logging is configured via the Management tab. The Management Tab, located on the Navigation bar, displays all menus related to device management. These menus appear in the Navigation tree and include the following: • Management Configuration • Software Update To configure the syslog settings, click on the management tab followed by the management Settings option (see figure above). Fill in the IP address and port of the external syslog server and change the Enable Syslog option to enable. At this point, logging messages should begin to appear on the external log collector. The level of logging may need to be adjusted depending upon the needs of the administrator. 13.1.1. Logging Levels Configuration Configuration of logging levels must be done from the Advanced Parameters Page. Open the 'Advanced Parameters' page (Configuration tab > Protocol Configuration menu > SIP Advanced Parameters submenu > Advanced Parameters page item). Under the CDR and Debug heading (see the figure below), the logging levels can be set as follows: • [0] = Debug is disabled (default). • [1] = Flow debugging is enabled. • [2] = Flow and device interface debugging are enabled. • [3] = Flow, device interface, and stack interface debugging are enabled. • [4] = Flow, device interface, stack interface, and session manager debugging are enabled. • [5] = Flow, device interface, stack interface, session manager, and device interface expanded debugging are enabled. Note that if traces are necessary, then the debug level is usually set to 5. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
87
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
88
13.2.
Internal Logging Syslog messages can also be viewed internally, i.e. on the Audio Codes device itself. 1. Set the logging debug levels as described in the previous section. 2. Open the Message Log Page (Status & Diagnostics Tab > Status & Diagnostics Menu > Message Log page item). 3. A logging page similar to the one below will open. Log messages are color coded as follows: a. Yellow‐ Fatal Error Message b. Blue‐ Recoverable Error Message c. Black‐ Notice Message 4. To clear the logging page, click Message Log again and the page will clear and receive new messages. 5. To stop the messaging log, simply access any other page in the web interface. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
89
14. Common Troubleshooting Scenarios 14.1.
IPSI Board Link Failure Below is an example of an IPSI board link failure. In this case, it is a network link failure that prevents the IPSI board from communicating with any other network devices, including CM and the external syslog server. Therefore no messages from the IPSI board will appear in the syslog. Engineers are left to determine through deductive reasoning what is actually happening. In this case, note that at 15:02:22, the S8500 (172.31.105.202), complains that it cannot reach the IPSI board (172.31.105.101). This is an indicator of link failure. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
90
14.2.
Network Time Protocol Blocked to IPSI Board In some cases a firewall on the client network or the firewall settings on the Communications Manager server itself may be blocking Network Time Protocol (NTP). This protocol uses UDP port 123 to allow time synchronization between devices. If this protocol is blocked, then the IPSI board will not be able to synchronize its clock to that of the CM server. This problem will not be readily apparent until the Communications Manager server is rebooted. Once a reboot occurs, the IPSI board will lose track of time and reset its clock to January 1 0:0:00. Note the logging examples below. Example number 1 shows the IPSI board (172.31.105.112) with the correct time. Example 1 2008‐09‐09 11:06:54 Local4.Debug 172.31.105.150 10:38:48.680 MNT_MSG message: BAD_PORT_NUM 2008‐09‐09 10:38:48 User.Error 172.31.105.210 not restart it. 2008‐09‐09 10:38:47 Cron.Info 172.31.105.210 2008‐09‐09 10:38:47 User.Error 172.31.105.210 not restart it. 2008‐09‐09 10:38:48 User.Error 172.31.105.210 not restart it. 2008‐09‐09 10:38:48 User.Error 172.31.105.210 not restart it. 2008‐09‐09 10:38:48 Local4.Debug 172.31.105.112 2008‐09‐09 10:38:48 Local4.Debug 172.31.105.112 App FW Version:<009><009>44 2008‐09‐09 10:38:49 Local4.Debug 172.31.105.112 Build Time:<009><009>Wed May 14 08:36:27 MDT 2008 2008‐09‐09 10:38:49 Local4.Debug 172.31.105.112 Build Workspace:<009>fisk__080514_ipsiV44sipiV14mklbl BTJ
AR RS
Sep 9 10:38:48 172.31.105.150 TN799[ tMtceMain svc_mon: no lock file found for /opt/ecs/sbin/hp‐sshd. will crond[25087]: (root) CMD (/usr/lib/sa/sa1 1 1) svc_mon: no lock file found for /opt/ecs/sbin/hp‐sshd. will svc_mon: no lock file found for /opt/ecs/sbin/hp‐sshd. will svc_mon: no lock file found for /opt/ecs/sbin/hp‐sshd. will Sep 9 10:38:48 TN2312[ 0: 01/01 00:00:00 (ttargetMain): Sep 9 10:38:48 TN2312[ 1: 01/01 00:00:00 (ttargetMain): Sep 9 10:38:48 TN2312[ 2: 01/01 00:00:00 (ttargetMain): Sep 9 10:38:48 TN2312[ 3: 01/01 00:00:00 (ttargetMain): Avaya Syslog Implementation Guide
TRS
91
In example number 2 below, NTP has been blocked and the CM server has been restarted. Note that the time on the IPSI board has not synchronized and has reverted to its default settings. Example 2 2008‐09‐09 16:44:11 Local4.Debug 172.31.105.151 Sep 9 16:16:00 172.31.105.151 TN2602[ 09 16:16:00.000 (HealthEngineTas): Peer bits = 0000 0000 0000 0000 2008‐09‐09 16:44:11 Local4.Debug 172.31.105.151 Sep 9 16:16:00 172.31.105.151 TN2602[ 09 16:16:00.000 (HealthEngineTas): Sending local MGR candidate update= 2 2008‐09‐09 16:44:45 Local4.Debug 172.31.105.112 Jan 0 00:00:33 TN2312[ 00 00:00:33.880 (SvrListMgr): updateServerStatus(): Server 1: CONNECTED+STANDBY+NO_TIMER => CONNECTED+ACTIVE+NO_TIMER 2008‐09‐09 16:44:45 User.Error 172.31.105.210 : >#2,YY,RES,01A,PKT‐
INT,y,MAJ,MAJ,N,09/09:16:16:34,none,none,0x0:0x4100:4137:31005:09/09:16:15:52$!# 2008‐09‐09 16:44:45 User.Error 172.31.105.210 : >#2,YY,RES,01A,PKT‐
INT,y,MAJ,MAJ,N,09/09:16:16:34,none,none,0x0:0x4100:4137:31005:09/09:16:15:52$!# 2008‐09‐09 16:44:45 Local4.Debug 172.31.105.112 Jan 0 00:00:34 TN2312[ 00 00:00:33.900 (tSimRead): simProcMsg(): Received SIM_CAPABILITY_EXCH message 2008‐09‐09 16:44:45 Local4.Debug 172.31.105.112 Jan 0 00:00:34 TN2312[ 00 00:00:33.900 (tSimRead): simProcMsg(): CM SIM_CAPABILITY_EXCH message contents, msgId=0x22<009>capability=0x1 2008‐09‐09 16:44:45 Local4.Debug 172.31.105.112 Jan 0 00:00:34 TN2312[ 00 00:00:33.900 (tSimRead): simProcMsg(): IPSI_CAPABLE = 0x3 2008‐09‐09 16:44:45 Local4.Debug 172.31.105.112 Jan 0 00:00:34 TN2312[ 00 00:00:33.900 (tSimRead): processServStatInd(): Got SERVER_STATUS_IND: ACTIVE, Serve BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
92
14.3.
IPSI Board Duplex Mismatches Duplex mismatches between the IPSI board and the data switch can cause checkslot errors. This scenario can take place if the duplex is mistakenly hard‐coded to half. The error may also take place if the IPSI board and the switch are not set up to negotiate properly and half duplex is defaulted to. The problem can occur if duplex on one side is hard‐coded with the other side set to auto‐negotiate. In this case the duplex cannot be negotiated and defaults to half duplex. Error Type: checkSlot: – Usually appended with some sort of “sanity failure” statement, these errors indicate that the communication between the IPSIs and the Servers has been interrupted. Often, this is due to a speed/duplex mismatch between the data‐switches and the IPSIs or Servers, which causes packet‐loss. Syslog example of duplex mismatch 20061024:151434782:1960:pcd(5127):MED:[[1:0] checkSlot: sanity failure (1)] 20061024:151435782:1961:pcd(5127):MED:[[1:0] checkSlot: sanity failure (2)] 20061024:151436782:1962:pcd(5127):MED:[[1:0] checkSlot: too many sanity failures] These entries may also be present indicating a server interchange or Port‐network restart after checkSlot sanity failures. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
93
15. Syslog Bandwidth Utilization Syslog traffic from Avaya products consumes a very small amount of bandwidth. A laboratory environment was constructed consisting of a Communications Manager 5.1 server, and a G650 gateway containing an IPSI board, CLAN, and Crossfire board. Syslogging was set for all of these devices to log to an external syslog server. Traffic was then captured via Clearsight Analyzer under normal conditions with no calls running. The following results were observed: IPSI Syslog Traffic 5 Kilobits/second Average
74 Kilobits/second Max CM Syslog Traffic 20 Kilobits/second Average
115 Kilobits/second Max CLAN Syslog Traffic 7 Kilobits/second Average
10 Kilobits/second Max Crossfire Syslog Traffic 15 Kilobits/second Average
138 Kilobits/second Max Total Observed Usage 47 Kilobits/second Average
337 Kilobits/second Max The figure below demonstrates how little bandwidth is consumed by syslog when compared to other traffic types in the lab scenario described above. Bandwidth Comparison BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
94
The above bandwidth statements were obtained with the IPSI and CM log settings illustrated below. IPSI Log Values CM Logging Levels change logging‐levels Page 1 of 2 Enable Command Logging? y Log Data Values: both When enabled, log commands associated with the following actions: add? y export? y refresh? y busyout? y get? y release? y campon‐busyout? y go? y remove? y cancel? y import? y reset? y change? y list? y save? y clear? y mark? n set? y disable? y monitor? y status? y display? y netstat? y test? y duplicate? y notify? y traceroute? y enable? y ping? y upload? y erase? y recycle? y BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
95
change logging‐levels Page 2 of 2 LOGGING LEVELS Log All Submission Failures: y Log PMS/AD Transactions: y Log IP Registrations and events: y Log CTA/PSA/TTI Transactions: y 15.1.
Bandwidth Utilization in a Loaded Environment The same testing procedures and configuration parameters was also used in a live environment with a Communications Manager server and port network with multiple IPSI’s, CLAN’s and Medpro’s. The system was in use at the time with between 20 and 40 simultaneous calls. The results were moderately higher, at approximately 225 kbs, but are still very tolerable for healthy networks. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
96
16. Syslog and Traffic Engineering Although the actual consumption of syslog traffic is very low under normal conditions, bursts of syslog traffic can go higher during times when a component is in an error state or the logging levels are set high (Debug, for example). For this reason, syslog traffic should have Quality of Service (QoS) settings applied that prevent it from consuming bandwidth needed by other VOIP devices. This configuration will vary with customer needs, and the actual network components involved. Additional planning may be necessary if a customer has multiple remote sites that require logging. Depending on customer needs and the bandwidth available, a centralized logging scenario similar to the illustration below may make sense. This model allows all log traffic to be collected at a central site. The centralized model allows engineers to more easily gain access to logs; maintenance/backup of the log collectors is also simplified. Centralized Model ESD GRO UNDJA CK
ESD GRO UNDJA CK
P OWE R
1
2
3
4
5
6
7
8
9
10
11
12
13
14
P OWE R
ESD GRO UNDJA CK
P OWE R
1
2
3
4
5
6
7
8
9
10
11
12
13
14
P OWE R
P OWE R
FAN O R PO WER FAIL
FAN O R PO WER FAIL
FAN O R PO WER FAIL
FAN AND PO WER OK
FAN AND PO WER OK
FAN AND PO WER OK
AC INPUT
AC INPUT
1
2
3
4
5
6
7
8
9
10
11
12
13
14
P OWE R
AC INPUT
DC IN PUT
DC IN PUT
DC IN PUT
ACTIVE RING
ACTIVE RING
ACTIVE RING
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
97
If, however, the customer has remote locations with limited bandwidth, then a distributed logging model may make sense. A distributed logging system eliminates the need to have log traffic traverse the wide area network (WAN) circuits that may not have extra bandwidth necessary to support the additional traffic. This model may make the log server administration more difficult since the log collectors will be located at the remote sites. Note that each site, including the main headquarters, has its own syslog server. Distributed Logging Headquarters
Communication
Manager
G650
Syslog Server
ESDG ROU ND JACK
POW ER
1
2
3
4
5
6
7
8
9
10
11
12
13
14
POW ER
FAN OR POWE R FA IL
FAN AND POW ER OK
AC INP UT
DC INP UT
A CTI VE RING
Li
m
ted
mi
Li
n
Ba
h
idt
dw
ite
d
Ba
n
dw
id
t
h
ESDG ROU ND JACK
ESDG ROU ND JACK
POW ER
1
2
3
4
5
6
7
8
9
10
11
12
13
14
POW ER
POW ER
FAN OR POWE R FA IL
FAN OR POWE R FA IL
FAN AND POW ER OK
FAN AND POW ER OK
AC INP UT
Communication
Manager
1
2
3
4
5
6
7
8
9
10
11
12
13
14
POW ER
AC INP UT
DC INP UT
DC INP UT
A CTI VE RING
A CTI VE RING
G650
Syslog Server
Communication
Manager
Remote Site
G650
Remote Site
BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
98
17. Logging With Kiwi Tools Kiwi Tools provides a syslogging service that can be downloaded free of charge. Although the free version lacks some of the feature enhancements of the full version, it is still very usable. Start by downloading and installing the Kiwi Syslog Daemon for Windows. The software can be found at www.kiwisyslog.com. There is a choice during installation to choose to start the syslog service automatically during the system start up, or to allow the user to start the service manually. Users should choose the option that fits their needs 17.1.
Collecting Logs Once the software is installed, start the service by choosing Start=>Programs=>Kiwi Enterprises=>Kiwi Syslog Daemon. A screen similar to the figure below will appear. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
99
At this point it is important to choose where the logs will be captured to. This can be set by choosing File=>Setup. The following screen should appear: BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
100
Highlight and click the Log to File icon, this allows the user to change what the log file is called and where it will be located. Additional items such as log rotation, and the actual format of the log file can be changed here as well. Once these options are set, Kiwi will capture syslog data from any device that has been configured to send syslog messages to it. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
101
17.2.
Viewing and Sending Log Files Once a location for the actual log file has been chosen via the above steps, viewing and sending the file is simple. The log file will be saved as a text file that can be opened by any text editor. The log file can also be sent as an email attachment (providing it does not exceed any email size requirements that may have been set). In the above example, the location was set as C:\Program Files\Syslogd\Logs\SyslogCatchAll.txt BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
102
The figure below illustrates the default view of a log file. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
103
17.3.
Rules and Filtering With Kiwi Syslog Kiwi allows users to create rules that contain specific filters and actions to be taken if defined conditions are met. Up to 100 rules can be set up within Kiwi. Each rule contains filters by which log messages are matched. Rules are run from the top of the list down. The order of the rules list can be adjusted as necessary. The filters contained within the rules are also run in the user‐defined order. If any of the filtering conditions are not met, then the program stops processing that rule and moves on to the next rule. Note, the Filtering option may only be available in the registered version of Kiwi Tools. If all filtering conditions are met, then the user defined action is applied to the filtered syslog traffic. By default, the initial setup contains a single rule named Default with no filters defined. This ensures all messages are passed. The two default actions of "Display" and "Log to file" are used. This ensures that by default, all messages are displayed and logged to a file called "SyslogCatchAll.txt" which is located in the \Logs directory of the Kiwi Syslog Daemon installation folder.
Following is a list of actions that can be applied once all filter conditions are met: Display Log to a file Forward to another host Play a sound Run an external program Send an email Send a syslog message Send pager or SMS message Send SNMP trap Stop message processing Run a script Example Setting Up a Rule and Simple Filter In this example, a rule will be set up to filter for POP3 and SMTP syslog messages. If a message is found, the rule will define an action of “log to file”. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
104
Begin adding a rule by choosing File=>Setup, right click Rules and chose “Add Rule”. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
105
For this example, a rule named Test Rule will be created. Once created, right click on Filters underneath the new rule and chose “Add Filter. The following screen appears informing the user that filed is blank and to chose the drop down menus above. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
106
Configure this filter to look for the text “POP 3” or “SMTP” anywhere in the message text portion of a syslog message. Chose Message text from the top left drop down box, leave the Type as “Simple”. More complex rules can be created if necessary. In this case a simple filter named POP 3 is searching the message text for the words “pop 3” or “SMTP”. Multiple search strings can be listed in the same line, all must be inside double quotes. The filter will match on any of the strings listed, utilizing an implicit OR. The [C] button selects a case sensitive or case in‐sensitive string search. Notice that the [C] button is pressed; this indicates that upper and lower case strings will be searched. The [S] button selects a sub‐string search or an exact, whole string match. Notice the [S] button is pressed, indicating a sub‐string search. This means the search strings can appear anywhere in the text. Click OK to save this filter. The next step is to define an action once criteria that meets the above filter is found. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
107
Now an action must be created for the filter matches. Right click on Action and choose Add Action. The following screen with a drop down box of choices appears. The user can then choose the action that best meets their needs. In this case Log to File was chosen. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
108
17.4.
Highlighting Text Within Kiwi Highlighting in Kiwi is an option that allows the user to highlight specific types of text within the Kiwi system display grid. These options are actually sets of rules that will run in the order listed in the Highlighting Options box. These rules are applied from the top down. Any rule matches that occur will have the given highlighting applied to the syslog message. Note, the highlighting option may only be available in the registered version of Kiwi Tools. Highlighting Configuration To begin, choose View =>Highlighting Options. Answer Yes if asked to create default highlighting rules. The Highlighting Options box should appear. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
109
This box lists the highlighting rules that will be applied to each syslog message that is to be displayed, the syslog message field that will be searched, the string pattern that will be searched for, and the effect to be applied. Each rule can be activated/deactivated by respectively checking/unchecking the checkboxes leftmost on each row of the list. The list of fields available in the 'fields' drop‐down box is the same as the fields that are available on the Kiwi Syslog main display grid. (ie. Date, Time, Priority, Hostname, Message). Highlighting rules can be added/deleted by clicking the buttons on the toolbar to the right of the highlights list. Rule precedence can be changed in this toolbar as well, by clicking the up/down arrows. Note: That the first time the Highlighting Options is accessed, the user may be prompted "No highlighting rules have been found. Do you want to create some default rules based on Syslog Priorities?". Answering yes to this question creates some default rules based on Syslog Priority. These default rules are shown above. BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
110
References Avaya, “Maintenance Procedures for Avaya Communication Manager 3.1, Media Gateways and Servers”, February 2006. Avaya “Modular Messaging 5.0 With MSS Messaging Application Server (MAS) Administration Guide”, January 2009. Avaya, Gateway SRAD, “Error Logging Gateway Media Gateway Process (MGP) and Voice‐ Over‐IP (Voip), Cliff Grimes, February 2008, CID: 123068 Avaya, “Avaya Call Manager System Security Whitepaper, Andy Zmolek, December 2007, CID: 122957. Avaya, “Syslog Feature Test Specificaiton 46xx Phones, Diane Somers, April, 2002, CID: 92501. Avaya, “Logging in CM 4.0”, Russell Singer, February, 2007 Avaya, TN Boards and Branch Gateway Functionality, Presented by Ashok Reddy National Institute of Standards and Technology (NIST), “Guide to Computer Security Log Management, Recommendations of the National Institute of Standards and Technology (NIST), Karen Kent and Murugiah Souppaya, September 2006, http://csrc.nist.gov/publications/nistpubs/800‐92/SP800‐92.pdf IETF “RFC 3164: The BSD Syslog Protocol”, www.ietf.org, August 2001 BTJ
AR RS
Avaya Syslog Implementation Guide
TRS
111
Open as PDF
Similar pages