Troubleshooting
Mashery Local provides a set of tools that help an administrator to debug issues with API calls as they flow through TIBCO Mashery, troubleshoot networking issues with the system, identify issues with cloud synchronization, and collect system logs to facilitate Operations and Support staff to identify root-cause faster. This section outlines the tools available and their usage scenarios.
Verbose Logs
The Mashery Local administrator can troubleshoot issues related to call payloads or identify any inconsistencies as API call data flows through TIBCO Mashery by enabling verbose logs on API calls.
This feature is not enabled as an “always on” feature as producing these verbose logs may have some impact on API call performance. Instead, options are provided on the Cluster Manager UI to enable and disable verbose logs.
Using the Verbose Logs Feature
To use the Verbose Log feature:
Procedure
1. Specify the Verbose Logs location.
a) Select Use NFS.
TIBCO Mashery highly recommends using NFS location for verbose logs so that local system usage is not impacted. In addition, all the nodes in a cluster (the Master and Slaves) can write to the same centralized location for easier, further analysis.
1. Enter the NFS host name.
2. Enter the NFS directory name.
3. Click Save.
The specified NFS directory is mounted onto
/mnt/nfs/home
local directory location.
42
TIBCO Mashery
®
Local Installation and Configuration Guide for Docker
43
b) If you do not select Use NFS (not recommended), verbose logs will be saved to the local directory. The default directory
</var/log/mashery/verbose>
can be changed.
TIBCO Mashery
®
Local Installation and Configuration Guide for Docker
44
2. Enable Verbose Logs.
a) Select duration for capturing the logs (05, 10, 15, or 30 minutes).
b) Click Enable.
TIBCO Mashery
®
Local Installation and Configuration Guide for Docker
45
After you enable verbose logs, TIBCO Mashery Local writes the call data logs that include inbound request data, inbound processed data, outbound response data, and outbound processed data. Verbose logs (call data capturing) is disabled after the selected time duration expires.
You must set the Verbose Logs Location on each node in the cluster including Master and all
Slaves. Enabling or disabling verbose logs can only occur on the Master node. The Slave nodes just inherit the current verbose log enablement status from the Master.
Working with Verbose Logs
A directory is created every minute with the name format as YYYY-MM-DD-HH-MM. All the calls that are logged in a minute become part of one directory and so on. For each call, a sub-directory is created using the name <timestamp>-<Mashery Message ID>.
Mashery Message ID is a globally unique ID generated for every API call that is processed by TIBCO
Mashery. The Mashery Message ID provides a possible mechanism for administrators to create a golden thread for debugging issues between TIBCO Mashery, your partners and your backend system.
To be able to include this GUID in request and response headers, you can toggle on Include X-
Mashery-Message-ID in Request and Include X-Mashery-Message-ID in Response properties on in the Services>Endpoint>Properties page in the TIBCO Mashery Administration Dashboard.
Within each sub-directory, four log files are
InboundRequest.log
,
InboundProcessed.log
,
OutboundResponse.log
,
OutboundProcessed.log
.
●
●
InboundRequest
contains the request data on the API call as it is originally received by Mashery
Local from the client application.
InboundProcessed
contains the Mashery processed version of the inbound request as sent to API server (or to cache if enabled).
TIBCO Mashery
®
Local Installation and Configuration Guide for Docker
46
●
●
●
●
●
●
●
●
●
●
OutboundResponse
contains the response data as it is originally received by Mashery from the API server (or from cache if enabled).
OutboundProcessed
contains the Mashery processed version of the outbound response as sent to the client application.
Each of the four files contain some important metadata written as key-value pairs with 1 pair on one line. After the metadata, a new delimiter line is written followed by the actual message.
The metadata included are:
Key: Key a developer uses to get access to a Package Plan or API.
Service ID: TIBCO Mashery generated unique ID to identify an API.
Endpoint ID: TIBCO Mashery generated unique ID to identify an Endpoint.
Site ID: TIBCO Mashery generated unique ID to identify your Site within the TIBCO Mashery
Network.
IP address: IP address of the client application invoking the API call.
Method (if available): Method that was being accessed in the API call (available if appropriate
Method Configuration settings are specified in Services>End-points>Properties tab in the TIBCO
Mashery Administration dashboard).
Cache hit: 1 if cache is enabled and response is met from cache, 0 otherwise.
Error message (if any): TIBCO Mashery generated error message on that API call (if any).
Mapping Endpoint IDs
TIBCO Mashery Local provides a script that allows fetching a list of endpoints with details such as the
Endpoint ID and the Endpoint name. The Endpoints associated with a service are displayed. The
Service ID is the parameter used to fetch the Endpoint details.
//Request searching with a particular service id python getEndpointNames.py --service 95bpf2jv3f8p5x3xqsu657x5
//Response in json formatter
{
"services":[
{
"endpoints":[
{
"id":"7xwgjatahmuwgrz79cgw286a",
"name":"CORS-disabled"
},
{
"id":"2m4zz8nw4n9w36uau7j2bnqb",
"name":"Custom CORS(custom rest as the API key source)"
},
{
"id":"g2qx6vhxubu4d4w66egqnxsh",
"name":"CORS-enabled- EIN-112-dontallow"
},
{
"id":"uavv2nm6yy7j94nhp8zp5kjf",
"name":"CORS-enabled-EIN-112"
},
{
"id":"pgbrzzu89dtyvumqht4ncnt4",
"name":"preflight-requestmultipledomainno"
},
{
"id":"7qcpe6dsss4kxp4u8c6fy5pr",
"name":"EIN-222"
}
],
"id":"95bpf2jv3f8p5x3xqsu657x5"
}
TIBCO Mashery
®
Local Installation and Configuration Guide for Docker
47
]
}
Debugging Utility
A debug utility is provided that can be used to capture information about system health and configuration, connectivity to the cloud for synchronization, fix Slave registration issues, and resolve any replication issues between Master and Slave. A command line console is available to run the various options available. This utility is useful for gathering information to assist with trouble-shooting common system configuration errors with TIBCO Mashery Local.
Running the Debug Utility
Execute the following command to run the debug utility:
$ python /opt/mashery/utilities/debug_util.py
The following options are available. Some options are available to be run only on Master and some only on Slave.
Select from the following:
1: Collect Logs
2: Test connectivity to Cloud Sync
3: Show Slave Status
4: Check IP address
5: Update record or Master IP address in Master (Master IP address has changed and registration of new Slave with cluster fails)
6: Fix slave corruption (Restart slave at last valid read position)
7: Update record of Master IP address in old Slave node (Master IP address has changed and cluster is not updated)
8: System manager (Remove non-functional or unused slaves form Master)
9: Collect system state (Disk health, process health, time setting, network settings) menu: Show this menu exit: Quit
For option 9: Collect system state, the resulting files for this option are created in the home directory, depending on the login users (root/administrator).
Collect Logs
This tool produces a tar.gz
file that collects Traffic Manager component logs, sync status with the cloud, the Slave and Master IP address checks, logs required to check replication issues between Master and Slave and verbose logs for the day (if any).
This option can be run on Master and Slave nodes.
Test Connectivity to Cloud Sync
This tool helps to determine if there are any errors connecting to the TIBCO Mashery Cloud system for synchronization.
This option can be run on Master and Slaves.
Show Slave Status
This option displays whether a Slave is functioning correctly, including its status, the Master systems IP address and any replication errors that are present between Master and Slave.
This option can be run on a Slave node.
Check IP Address
This option allows you to check the current IP address of the Master.
TIBCO Mashery
®
Local Installation and Configuration Guide for Docker
This option can be run on a Master node.
Update Record of Master IP Address in Master
Sometimes if the IP address of a Master node changes, new Slave registration with the Master fails.
Running this option fixes the record of the Master IP address in the Master node for successful Slave registration.
This option can be run on a Master node.
Fix Slave Corruption
This option allows you to resolve Master Slave replication issues.
This option can be run on a Slave node.
Update Record of Master IP Address in Old Slave Node
This option updates the record of the Master IP address in the Slave nodes and is useful for resolving
Master-Slave replication issues.
System Manager (Remove Non-functional or Unused Slaves from Master)
Sometimes Slave nodes are decommissioned and new Slave nodes are created. This option on Master system can be used to remove unused slaves.
System Level Troubleshooting
TIBCO Mashery Local administrators have the ability to run the following select commands to investigate and troubleshoot the network or system level issues.
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ping ping6 tracepath tracepath6 tcpdump traceroute sudo arping sudo tshark sudo route sudo ifconfig sudo iptables sudo dhclient sudo pvresize sudo resize2fs sudo edit for the following files:
/etc/resolv.conf
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network-scripts/ifcfg-eth1
48
TIBCO Mashery
®
Local Installation and Configuration Guide for Docker
●
●
●
●
/etc/rc.local
/etc/hosts
/etc/securitylimits.conf
/etc/nssswitch.conf
49
TIBCO Mashery
®
Local Installation and Configuration Guide for Docker
