EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE
INSIGHTIQ DOES NOT DISPLAY
DATA CORRECTLY IN THE FILE
SYSTEM REPORTING PAGE
Abstract
This guide helps you troubleshoot problems with displaying reports on the
InsightIQ File System Reporting page.
December 28, 2015
1 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Contents and overview
Note
Follow all of these steps, in order, until you reach a resolution.
1. Follow these steps.
2. Perform troubleshooting steps in order.
3. Appendices
Before you begin
Page 3
Start troubleshooting - initial diagnosis
Page 4
All reports are blank
Page 5
Server error page is displayed
Page 9
Missing report dates, or no reports at all
Page 11
Appendix A :
If you need further assistance
Appendix B :
How to use this flow chart
2 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Before you begin
CAUTION!
If the node, subnet, or pool you are working on goes down during the course of troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability.
Therefore, make sure you have more than one way to connect to the cluster before you start this troubleshooting process. The best method is to have a serial cable available.
That way, if you are unable to connect through the network, you will still be able to connect to the cluster physically.
For specific requirements and instructions for making a physical connection to the cluster, see article 16744 on the EMC Online Support site.
Before you begin troubleshooting, confirm that you can either connect through another subnet or pool, or that you have physical access to the cluster.
Configure logging through SSH
We recommend configuring screen logging to log all session input and output on both the Isilon cluster and the InsightIQ server instance during your troubleshooting session. These log files can be shared with EMC Isilon Technical Support if you require assistance at any point during troubleshooting.
Configure logging on the Isilon cluster
Note: The screen session capability does not work in OneFS 7.1.0.6 and 7.1.1.2. If you are running either of these versions, please configure logging using your local SSH client's logging feature.
1. Open an SSH connection to the cluster and log in using the root account . Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be preceded by the sudo prefix.
2. Change the directory to /ifs/data/Isilon_Support by running: cd /ifs/data/Isilon_Support
3. Run the following command to capture all input and output of the session : screen -L
This will create a file called screenlog.0 that will be appended to during your session.
4. Perform troubleshooting.
Configure logging on the InsightIQ server instance
1. Open an SSH connection to the InsightIQ server instance and log in by using the administrator account .
2. Run the following command to capture all input and output of the session : screen -L
This will create a file called screenlog.0 that will be appended to during your session.
3. Perform troubleshooting.
3 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Start troubleshooting - initial diagnosis
Introduction
Start troubleshooting here. If you need help understanding the flow chart conventions used in this guide, see How to use this flow chart .
Start
If this is your issue...
InsightIQ 3.2 only:
No File System Analytics (FSA) reports are visible after upgrading or installing the InsightIQ 3.2 server.
Reports that existed prior to the upgrade are visible.
Do this...
This is a known issue. Go to:
InsightIQ 3.2 server cannot view FSA reports
generated on the clusters, article 206108 .
You cannot view Quota Reports because you do not have permission.
Go to:
The InsightIQ user on the cluster does not have permission to view quota reports
, article 194112 .
All reports are blank.
You see the report date listed, but when you run the report, no data displays.
Server Error page appears.
A Server Error page (orange and black screen) appears when you attempt to view data.
Missing report dates, or no reports at all.
Go to
Go to
Go to
Page 5
Page 9
Page 11
.
.
.
Any other issue, including specific problems with the Capacity, Deduplication, or Quota
Reports.
Note the page number that you are currently on.
Upload log files and contact Isilon Technical Support, as instructed in Appendix A .
4 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
All reports are blank
You could have arrived here from:
Page 4 - Start
Page
5
If you have not done so already, configure logging through SSH, as described on Page 3 .
Open an SSH connection to the InsightIQ virtual machine and log on by using the administrator account.
Check whether the /ifs/.ifsvar/modules/fsa export is mounted.
Run the following command: mount -v
The output displays all of the mounted file systems.
InsightIQ 3.1 and later
If the /ifs/.ifsvar/modules/fsa export is mounted, InsightIQ can access the fsa directory, and the output looks similar to the following:
<cluster_IP_address>:/ifs/.ifsvar/modules/fsa on /mnt/<cluster_IP_address>/ifs/.ifsvar/ modules/fsa type nfs (rw,nosuid,nodev,intr,sloppy,addr=<cluster_IP_address>)
InsightIQ 3.0 and earlier
If the /ifs/.ifsvar/modules/fsa export is mounted, InsightIQ can access the fsa directory, and the output looks similar to the following:
<cluster_IP_address>:/ifs/.ifsvar/modules/fsa on /net/<cluster_IP_address>/ifs/.ifsvar/
modules/fsa type nfs (rw,nosuid,nodev,intr,sloppy,addr=<cluster_IP_address>
Yes
Is the export mounted?
No
Go to Page 7
Go to Page 6
5 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
All reports are blank, continued
The export is not mounted
You could have arrived here from:
Page 5 - All reports are blank
Page
6
Attempt to mount the export. For instructions, see section
"Solution 3. Disable automount, recreate the InsightIQ path, and mount that path" of the following article:
FSA reports fail when InsightIQ cannot access the FSA
database, 88945
Yes
Is the export mounted now?
No
Refresh the InsightIQ File
System Reporting page.
Note the page number that you are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
Yes
Do the reports display correctly now?
No
End troubleshooting
Go to Page 7
6 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
All reports are blank, continued
Export is mounted
You could have arrived here from:
Page 5 - All reports are blank
Page
7
CHECK FOR A CLUSTER PERMISSIONS ISSUE
The InsightIQ user might not have permission to access files all the way through the path.
Copy and paste the following command into the InsightIQ command-line interface to see the permissions on all of the clusters that InsightIQ is monitoring .
Note: When you copy and paste the command into the command-line interface, the command will appear on multiple lines (exactly as it appears here), but when you press Enter, the command will run as it should.
InsightIQ 3.1 and later
for i in $(ls -1d /mnt/*); do ls -ld $i; ls -ld $i/ifs; ls -ld $i/ifs/.ifsvar; \ ls -ld $i/ifs/.ifsvar/modules; ls -ld $i/ifs/.ifsvar/modules/fsa; done
InsightIQ 3.0 and earlier
for i in $(ls -1d /net/*); do ls -ld $i; ls -ld $i/ifs; ls -ld $i/ifs/.ifsvar; \ ls -ld $i/ifs/.ifsvar/modules; ls -ld $i/ifs/.ifsvar/modules/fsa; done
Example output (note that for InsightIQ 3.0 and earlier, the output file path will be /net* instead of /mnt*): drwxr-xr-x 3 root root 0 Feb 10 20:42 /mnt/cluster drwxr-xr-x 5 root root 0 Feb 10 20:42 /mnt/cluster/ifs drwxr-xr-x 3 root root 0 Feb 10 20:42 /mnt/cluster/ifs/.ifsvar
drwxr-xr-x 3 root root 0 Feb 10 20:42 /mnt/cluster/ifs/.ifsvar/modules drwxr-xr-x 4 root root 97 Feb 11 07:30 /mnt/cluster/ifs/.ifsvar/modules/fsa
Do all directories have permissions of at least
755 (
rwxr-xr-x)?
Yes
No
Add permissions for the directories with fewer than 755 permissions. Log in to the cluster command-line interface as the root user and run the following command for each directory with fewer than 755 permissions: chmod 755 <path to directory>
Go to Page 8
Go to Page 8
7 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
All reports are blank, continued
The export is mounted
You could have arrived here from:
Page 7 - All reports are blank
Page
8
Refresh the InsightIQ
File System Reporting page.
Yes
End troubleshooting
Does the report display correctly?
No
Note the page number that you are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
8 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Server Error page is displayed
You could have arrived here from:
Page 4 - Start
Page
9
If you have not done so already, configure logging through SSH, as described on Page 3 .
Are you running
InsightIQ 2.5?
Yes
This is a known issue, which is fixed in InsightIQ 3.0.
Upgrade to to InsightIQ 3.0 or later.
No
Go to Page 10
Do the reports display correctly now?
Yes
End troubleshooting
No
Go to Page 10
9 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
File System Reporting page, continued
Gather InsightIQ logs and debug information
You could have arrived here from:
Page 9 - Server Error page is displayed
Page
10
Contact Isilon Technical Support to open a
Service Request (SR). Make note of your
SR number. You will need it to perform the following steps.
On the InsightIQ command-line interface, perform the following steps to collect debug information for Isilon Technical Support to use for further troubleshooting :
1. Open /etc/isilon/insightiq.ini in a text editor.
2. Edit the line under [DEFAULT] so that debug = true:
[DEFAULT] debug = true
3. Restart InsightIQ: iiq_restart
4. Refresh the error page. The server error and the debug information will appear.
5. Copy any information displayed on the screen into your Service Request .
6. Open /etc/isilon/insightiq.ini in a text editor.
7. Edit the line under [DEFAULT] so that debug = false:
[DEFAULT] debug = false
8. Restart InsightIQ: iiq_restart
Note the page number that you are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
10 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Missing report dates, or no reports at all
You could have arrived here from:
Page 4 - Start
Page
11
If you have not done so already, configure logging through SSH, as described on Page 3 .
Attempt to mount the export. For instructions, see section
"Solution 3. Disable automount, recreate the InsightIQ path, and mount that path" of the following article:
FSA reports fail when InsightIQ cannot access the FSA
database, article 88945
Yes
Is the export mounted now?
Go to Page 12
No
Note the page number that you are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
11 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Missing report dates, or no reports at all, continued
You could have arrived here from:
Page 11 - Missing report dates, or no report at all
Page
12
Refresh the InsightIQ
File System Reporting page.
Yes
End troubleshooting
Do the reports display correctly now?
No
Go to Page 13
12 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Missing report dates, or no reports at all, continued
You could have arrived here from:
Page 12 - Missing report dates, or no report at all
Page
13
From the OneFS command-line interface, run the following command for your version of OneFS:
OneFS 7.1 and later
isi job events list --job-type=fsanalyze
OneFS 6.5 and 7.0:
isi job history -j fsanalyze
Note
The report date listed in
InsightIQ report selection dropdown menu is the date that the FSAnalyze job started.
If there are missing reports, does the output from the previous step show that an FSAnalyze job completed successfully on the dates for which there are missing reports?
If there are no reports, does the output show that an FSAnalyze job has ever completed?
Yes No
Go to Page 14
Note the page number that you are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
13 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Missing report dates, or no reports at all, continued
You could have arrived here from:
Page 13 - Missing report dates, or no report at all
Page 15 - Missing report dates, or no report at all
Page 16 - Missing report dates, or no report at all
Page
14
On the InsightIQ command-line interface, check whether the FSA reports exist. Run the following command, where <cluster> is the name configured as the OneFS host in the InsightIQ web interface.
(To find the name configured as the OneFS host: From the InsightIQ web interface, click Settings > Monitored Clusters. For the cluster you want, in the Actions column, click Configure. In the Cluster Credentials tab, see the OneFS host field.)
InsightIQ 3.1 and later:
[root@iiq ~]# find /mnt/<cluster>/ifs/.ifsvar/modules/fsa/pub/ -name "*.db" -exec ls -la {} \;
InsightIQ 3.0 and earlier:
[root@iiq ~]# find /net/<cluster>/ifs/.ifsvar/modules/fsa/pub/ -name "*.db" -exec ls -la {} \;
The output lists the permissions and size of each report for a year (default). Make sure the command returns at least one
results.db result. If reports are missing only for certain dates, look through the list and see if there is a results.db result for those dates.
Example output:
-rw-r--r-- 1 root root 5898240 Feb 8 06:12 /net/cluster/ifs/.ifsvar/modules/fsa/pub/job.221/results.db
-rw-r--r-- 1 root root 5898240 Feb 7 06:14 /net/cluster/ifs/.ifsvar/modules/fsa/pub/job.219/results.db
-rw-r--r-- 1 root root 5898240 Feb 6 06:15 /net/cluster/ifs/.ifsvar/modules/fsa/pub/job.217/results.db
Yes
Go to Page 15
Is at least one
results.db result returned?
No
Note the page number that you are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
14 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Missing report dates, or no reports at all, continued
Page
15
You could have arrived here from:
Page 14 - Missing report dates, or no report at all
Check to see whether all of the required ports are enabled on your network. For a list of required ports, see the "InsightIQ system requirements - OneFS" section of the InsightIQ Installation Guide for your version of InsightIQ.
Are all of the required ports enabled?
Yes
No
Enable the ports.
Go to Page 16
Go back to Page 14
15 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Missing report dates, or no reports at all, continued
You could have arrived here from:
Page 15 - Missing report dates, or no report at all
Page
16
Check the output from Page 14 to see if the permissions for each item in the output are at least 644 (
rw-r--r--).
Are the permissions for each item in the output at least 644?
Yes
Go to Page 18
No
Go to Page 17
16 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Missing report dates, or no reports at all, continued
You could have arrived here from:
Page 16 - Missing report dates, or no report at all
Page
17
Add permissions for the files with fewer than 644 permissions. Log in to the cluster command-line interface as the root user and run the following command for each file with fewer than 644 permissions: chmod 644 <path to file>
Yes
End troubleshooting
Refresh the InsightIQ
File System Reporting page.
Can you view the FSA reports now?
No
Go to Page 18
17 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Missing report dates, or no reports at all, continued
You could have arrived here from:
Page 16 - Missing report dates, or no report at all
Page 17 - Missing report dates, or no report at all
Page
18
Wait for the next FSA job to complete
(the job runs daily by default).
Does the problem reoccur after the FSA job runs?
Yes
Note the page number that you are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
No
End troubleshooting
18 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Appendix A: If you need further assistance
Contact EMC Isilon Technical Support
If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with.
This information and the log file will help Isilon Technical Support staff resolve your case more quickly.
Gather and upload InsightIQ and OneFS log files and screen sessions
Follow the steps in the flow chart below.
Step 1: Open a Service Request (SR)
Contact Isilon Technical Support to open a service request.
Make note of your SR number. You will need it in the next step.
Step 2: Copy InsightIQ logs and screen log file to the Isilon cluster
1. In the InsightIQ screen session: When troubleshooting is complete, type exit to end your screen session.
2. Transfer the screen session to the cluster by running the following command , where <node_ip> is the IP address of the node you want to upload the logs to: scp screenlog.0 root@<node_ip>:/ifs/data/Isilon_Support/screenlog.iiq.txt
3. On the InsightIQ VM instance: Run the following commands to gather InsightIQ configuration information: cat /etc/isilon/insightiq.ini |grep api_username > ~/local_config.txt
cat /var/cache/insightiq/datastore.pickle >> ~/local_config.txt
ifconfig > ~/ifconfig.txt
mount > ~/mount.txt
rpm -q isilon-insightiq.x86_64 > ~/iiqversion.txt
4. Run the following command to compress the files generated by the previous commands , as well as the contents of the
/var/log directory, where <SR> is your Isilon Technical Support service request number: sudo tar -czvf ~/<SR>.tgz /var/log local_config.txt ifconfig.txt mount.txt iiqversion.txt
5. Copy the compressed file to the /ifs/data/Isilon_Support directory on the monitored cluster by using the scp (secure copy) command, where <SR> is the service request number, and <node_ip> is the node IP address used above: scp ~/<SR>.tgz root@<node_ip>:/ifs/data/Isilon_Support
Go to the next page
19 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Appendix A: If you need further assistance, continued
Gather and upload InsightIQ and OneFS log files and screen sessions, continued
You could have arrived
here from:
Appendix A
Continued from previous page
Step 3: Upload the Isilon node logs, screen log files, and InsightIQ logs to Isilon Technical Support
1. In the OneFS screen session: When troubleshooting is complete, type exit to end your screen session.
2. On the Isilon cluster: Open an SSH connection to the same node IP address where you copied files in the previous steps.
3. Gather and upload the node and InsightIQ logs, and include the SSH screen log files, by using the command appropriate for your method of uploading files. Replace <SR> in the command with your service request number. If you are not sure which method to use, use FTP. Note: For each method, there is one long command. When you copy and paste the command into the command-line interface, it will appear on multiple lines (as shown here), but when you press Enter, the command will run properly.
ESRS:
isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support/screenlog.0 \
-f /ifs/data/Isilon_Support/screenlog.iiq.txt \
-f /ifs/data/Isilon_Support/<SR>.tgz
FTP:
isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support/screenlog.0 \
-f /ifs/data/Isilon_Support/screenlog.iiq.txt \
-f /ifs/data/Isilon_Support/<SR>.tgz
HTTP:
isi_gather_info --http --local-only -f /ifs/data/Isilon_Support/screenlog.0 \
-f /ifs/data/Isilon_Support/screenlog.iiq.txt \
-f /ifs/data/Isilon_Support/<SR>.tgz
SMTP:
isi_gather_info --email --local-only -f /ifs/data/Isilon_Support/screenlog.0 \
-f /ifs/data/Isilon_Support/screenlog.iiq.txt \
-f /ifs/data/Isilon_Support/<SR>.tgz
SupportIQ:
isi_gather_info --local-only -f /ifs/data/Isilon_Support/screenlog.0 \
-f /ifs/data/Isilon_Support/screenlog.iiq.txt \
-f /ifs/data/Isilon_Support/<SR>.tgz \
--noupload --symlink /var/crash/SupportIQ/upload/ftp
3. If you receive a message that the upload was unsuccessful , refer to article 16759 on the EMC Online Support site for directions for uploading files over FTP.
Continue to troubleshoot your issue with Isilon Technical Support .
20 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Appendix B: How to use this flow chart
Introduction
Describes what the section helps you to accomplish.
You could have arrived here from:
Page # - "Page title"
Page
#
Directional arrows indicate the path through the process flow.
Yes
Decision diamond
Note
Provides context and additional information. Sometimes a note is linked to a process step with a colored dot.
No
Process step
CAUTION!
Caution boxes warn that a particular step needs to be performed with great care, to prevent serious consequences.
Process step with command: command xyz
Go to Page #
Optional process step
End point
Document Shape
Calls out supporting documentation for a process step. When possible, these shapes contain links to the reference document.
Sometimes linked to a process step with a colored dot.
21 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
Copyright © 2015 EMC Corporation. All rights reserved. Published in USA.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
EMC², EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners.
For the most up-to-date regulatory document for your product line, go to EMC Online Support
( https://support.emc.com
).
22 - EMC Isilon Customer Troubleshooting Guide: InsightIQ Does Not Display Data Correctly in the
File System Reporting Page
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback .
