Ascolog Insight User Manual
पीडीएफ
दस्तावेज़
विज्ञापन
विज्ञापन
Ascolog Insight 1.8 Managing Logs User Manual Company Ascolog GmbH Address A - 1220 Vienna, Handel-Mazzetti-Gasse 2A Phone +43 680 2085161 E-Mail [email protected] Author Support Publication date 2014-11-08 Document Version 1.8 Copyright © 2011-2014 Ascolog GmbH Permission is granted to use, copy, and distribute this documentation provided that the copyright notice and this paragraph appear in all copies. All trademarks within this guide belong to their legitimate owners. 2/147 Table of Contents 1 Introduction............................................................................................................................... 6 1.1 About this Documentation (!).................................................................................................. 6 1.2 What is Ascolog Insight (!)....................................................................................................... 6 1.3 Help....................................................................................................................................... 7 2 Installing Ascolog Insight........................................................................................................... 8 2.1 Supported Operating Systems................................................................................................. 8 2.2 License (!)............................................................................................................................... 8 2.3 How to Install (!)..................................................................................................................... 9 2.3.1 Installing with XCOPY (!)..................................................................................................... 10 2.3.2 Installing with the Setup (!)................................................................................................. 10 2.4 How to Uninstall................................................................................................................... 11 2.4.1 Uninstalling an XCOPY Installation......................................................................................11 2.4.2 Uninstalling a Setup Installation......................................................................................... 11 3 User Interface.......................................................................................................................... 12 3.1 Workspace (!)........................................................................................................................ 12 3.2 Menu, Toolbar and Context Menus........................................................................................ 13 3.3 Windows............................................................................................................................... 13 3.3.1 Log Window....................................................................................................................... 13 3.3.1.1 Configuring Columns...................................................................................................... 13 3.3.1.2 Log Record Selection....................................................................................................... 15 3.3.1.3 Plain Text....................................................................................................................... 15 3.3.2 Functions Window.............................................................................................................. 16 3.3.3 Details Window.................................................................................................................. 16 3.3.4 Graph Window................................................................................................................... 17 3.3.5 Files Window...................................................................................................................... 17 3.3.6 Clipboard Window.............................................................................................................. 20 3.4 Checking the Application's Status.......................................................................................... 20 3.5 Sorting Log Records.............................................................................................................. 20 3.6 Mapping Data onto Descriptions and Icons............................................................................21 4 Working with Logs and Layouts................................................................................................ 24 4.1 An Introduction to Logs and Layouts (!).................................................................................24 4.2 Working with Existing Layouts (!)........................................................................................... 26 4.2.1 Saving a Layout (!).............................................................................................................. 27 4.2.2 Editing a Layout................................................................................................................. 27 4.2.3 Sample Layouts Delivered with Ascolog Insight (!)...............................................................28 4.2.3.1 Plain Text....................................................................................................................... 28 4.2.3.2 Layouts used in the Video Tutorials.................................................................................28 4.2.3.3 Apache Logging Services™.............................................................................................. 29 4.2.3.4 XML based log formats................................................................................................... 29 4.2.3.5 Microsoft Windows® Event Logs......................................................................................31 4.2.3.6 Windows® Installer (MSI)................................................................................................. 33 4.2.3.7 OutputDebugString......................................................................................................... 33 4.2.3.8 Syslog............................................................................................................................. 34 3/147 4.2.3.9 Layouts for HTML and HTML Report................................................................................35 4.2.3.10 Timestamp................................................................................................................... 35 4.3 Creating a Layout with the Wizard (!).....................................................................................36 4.4 Creating a Layout without the Wizard....................................................................................40 4.4.1 Creating the Files of the Layout.......................................................................................... 40 4.4.2 Creating a Grammar – A Short Introduction .......................................................................42 4.5 Loading, Merging, Browsing and Closing Logs (!)...................................................................44 4.5.1 Merging Logs of Different Types........................................................................................ 46 4.6 Reset.................................................................................................................................... 46 5 Analyze Functions.................................................................................................................... 47 5.1 An Introduction to Analyze Functions (!)................................................................................47 5.2 Working with the Functions Tree (!).......................................................................................49 5.3 How to create Analyze Functions (!)....................................................................................... 51 5.4 Filter Expressions (!)............................................................................................................. 53 5.4.1 Wildcards........................................................................................................................... 53 5.4.2 Timeframe......................................................................................................................... 54 5.4.3 RANGE............................................................................................................................... 54 5.4.4 Generic Filter Expressions.................................................................................................. 55 5.4.5 Components Tab............................................................................................................... 55 5.4.6 Occurrence Tab................................................................................................................. 56 5.4.7 Operators in Filter Expressions........................................................................................... 57 5.5 Analyze Functions based on Data Groups (!)..........................................................................58 5.6 Color Rules........................................................................................................................... 59 5.7 Record Filter......................................................................................................................... 61 5.8 Tabs..................................................................................................................................... 62 5.8.1 Switching between Log Records in Different Tabs...............................................................63 5.8.2 Editing a Tab..................................................................................................................... 65 5.8.3 Removing Tabs.................................................................................................................. 68 5.8.4 Modifying the Tab list........................................................................................................ 69 5.9 Bookmarks........................................................................................................................... 70 5.10 Graphs............................................................................................................................... 71 5.10.1 Timeline Graphs.............................................................................................................. 72 5.10.2 Column Graphs................................................................................................................ 72 5.10.3 Graph Window Properties................................................................................................. 73 5.10.4 Graph Properties.............................................................................................................. 78 5.10.5 Zooming.......................................................................................................................... 80 5.10.6 Adjusting Graphs............................................................................................................. 81 5.10.7 Saving a Graph to a File.................................................................................................... 82 5.10.8 Fixed Samples.................................................................................................................. 82 5.10.9 Troubleshooting.............................................................................................................. 86 5.11 Data Extraction (!)............................................................................................................... 86 5.11.1 Extract Text (!)................................................................................................................. 86 5.11.2 Extract Column (!)............................................................................................................ 89 6 Searching and Monitoring Log Records.....................................................................................93 6.1 Searching............................................................................................................................. 93 4/147 6.2 Monitoring - Find in Files...................................................................................................... 94 7 Options................................................................................................................................... 97 8 Refreshing the Log Window...................................................................................................... 98 8.1 How to Keep the Displayed Logs Up-to-date.........................................................................98 8.2 Refresh Options.................................................................................................................... 99 8.3 How to Test Refreshing....................................................................................................... 100 9 Task Scheduling..................................................................................................................... 101 9.1 An Introduction to Task Scheduling.....................................................................................101 9.2 Scheduled Tasks Dialog...................................................................................................... 101 10 Custom Columns Definitions (CCD)......................................................................................103 10.1 Working with Custom Columns Definitions........................................................................103 10.2 Predefined Custom Columns............................................................................................. 107 10.2.1 Custom Columns Overview............................................................................................107 10.2.2 Performance Analysis of A Program................................................................................109 10.2.3 Visualizing Call Sequences of Functions -Enter/Exit Logging..........................................110 10.3 Controlling Custom Columns using a Graphical User Interface...........................................112 10.3.1 Correcting Timestamps (Time Shifting)...........................................................................112 11 Merging Logs -Advanced Topics..........................................................................................122 11.1 How Merged Logs are Created........................................................................................... 122 11.2 Changing the Merge Order................................................................................................122 11.3 Merging Records with Custom Columns............................................................................123 11.3.1 Merge Records and Recalc Records.................................................................................123 11.3.2 Options......................................................................................................................... 124 12 Layouts – Advanced Topics.................................................................................................. 126 12.1 Layout Files...................................................................................................................... 126 13 Grammars -Advanced Topics...............................................................................................128 13.1 How Grammars are used in Ascolog Insight.......................................................................128 13.2 Grammar Definition Dialog................................................................................................130 13.2.1 An Introduction to the Grammar Definition Dialog..........................................................130 13.2.2 Terminals...................................................................................................................... 136 13.2.2.1 Ambiguous Regular Expressions.................................................................................139 13.2.2.2 The Problem with Ambiguous Regular Expressions......................................................139 13.2.3 Nonterminals and Production Rules................................................................................140 14 Exporting Logs.................................................................................................................... 143 14.1 An Introduction to Exporting Logs.....................................................................................143 14.2 Conversion Dialog............................................................................................................. 143 14.3 XML.................................................................................................................................. 144 14.4 HTML................................................................................................................................ 145 14.5 HTML Report..................................................................................................................... 147 14.6 Syslog............................................................................................................................... 154 14.6.1 Configuring the Syslog Server........................................................................................ 154 14.6.2 Configuring the Syslog Client.........................................................................................156 15 Ascolog Insight as a Windows® service.................................................................................161 16 Ascolog Insight's Software Development Kit (SDK)................................................................162 5/147 1 Introduction 1.1 About this Documentation (!) In order to facilitate your orientation in this document the headers of very important chapters are highlighted in blue and followed by an exclamation mark. These chapters will explain the basic functionality which you will need to know when you want to analyze logs wit Ascolog Insight. How to work with Ascolog Insight as a user is the focus of this documentation. First the major parts of the user interface are introduced. The chapters 4 Working with Logs and Layouts and 5 Analyze Functions cover the most important features users need to know when working with Ascolog Insight. • User Interface elements are documented in bold (e.g. OK button, Tools menu). • For Ascolog Insight terminology an italic font is used (e.g. analyze function, layout). • Capital letters denote Ascolog Insight columns which contain a certain type of data from the log records (e.g. LOGTYPE, TIMESTAMP) • Strings, files, folder and paths are encompassed by quotation marks (e.g. “C:\ProgramData\AscologInsight”, “Warning”) 1.2 What is Ascolog Insight (!) Ascolog Insight is a software application for analyzing logs. The tool is a great help for software developers, administrators, quality assurance and support staff to understand the behavior of a software system based on its logs. It supports you to find and identify problems, errors and security issues. The program uses optimized algorithms to achieve unmatched processing performance even with very large logs. Grammar based parsing in combination with a specialized script language allows the processing of a wide variety of different log formats. Ascolog Insight provides a wizard which can automatically create a grammar for your log type. Furthermore an SDK is available to allow access to any kind of data source (e.g. binary files, databases, cloud storage). Ascolog Insight can also be run as a Windows® service. Purchasing an Ascolog Insight Professional license does not entitle you to use the Ascolog Insight SDK or to run Ascolog Insight as a Windows® service. If you are interested in this functionality please contact us to learn more or to get a quote. Please note that although this user manual is about Ascolog Insight Professional it will make references to running Ascolog Insight as a Windows® service. 6/147 1.3 Help In order to open the help go to the Help menu and select Help Topics or press the F1 button. This will open the Help dialog as shown in Figure 1.1. Figure 1.1: Help If you don't see the table of contents on the left side of the Help dialog press the Show button in the toolbar of the Help dialog or open the Options menu and select Show Tabs. 7/147 2 Installing Ascolog Insight 2.1 Supported Operating Systems Ascolog Insight is available as 32-bit and 64 bit version. The following operating systems are supported: • Windows Server 2003 • Windows Vista • Windows 2008 • Windows 7 • Windows 2008 R2 • Windows 8 2.2 License (!) A license issued by Ascolog GmbH is required to use the software. After the software is purchased the customer is provided with the license. However, for evaluation purposes you can also get a free evaluation license which expires after some time. A license consists of the owner name (optional), the company name (optional) and a license key. In order to use the software this license data must be entered in a dialog that pops up when the application is started for the first time (see Figure 2.1). Figure 2.1: License Information The next time the application is started the license data does not have to be provided again. In order to change a license open the Help menu and select Change License... This will also open the dialog shown in Figure 2.1 and you can enter the name, company name and license key of the new license. 8/147 The available functionality of Ascolog Insight depends on the type of the license. A license also has a serial number and it might specify an expiration date. You can check out your license information in the About Ascolog Insight dialog in the Help menu (see Figure 2.2). Figure 2.2: Version and license information 2.3 How to Install (!) There are two methods to install Ascolog Insight: • XCOPY - Just extract Ascolog Insight from a ZIP archive to a folder of your choice where you have read and write access. All files will be located within this folder. • SETUP - Run a setup application which installs Ascolog Insight and creates a shortcut in the start menu. Ascolog GmbH digitally signs all executable files (e.g. setup, application, self extracting archives) and libraries to ensure a maximum of safety. The name used for signing is Ascolog GmbH. The digital signature is displayed during the installation process or on the Properties page of a signed file (as shown in Figure 2.3). Please always check the digital signature. Consult your Microsoft Windows® documentation for more information on digital signatures. 9/147 Figure 2.3: Digital signature of the Ascolog GmbH 2.3.1 Installing with XCOPY (!) The ZIP archive consists of a folder called “AscologInsight” which contains all files of Ascolog Insight. Extract the ZIP archive to a folder of your choice where you have read and write access. There are no files that will be stored outside of the “AscologInsight” folder. If you want to use another location for Ascolog Insight just move the “AscologInsight” folder to this location and Ascolog Insight will still work. You can also rename the “AscologInsight” folder if you want. All files (program, data like layouts or sample log files) will be stored inside the Ascolog Insight folder. The data will be stored in a sub-folder called “DATA”. This folder will be referred to as data folder in this documentation. 2.3.2 Installing with the Setup (!) When you select this method Ascolog Insight will be installed in the way you are used to from other setup applications on the Microsoft Windows® operating system. Use this installation method if you want to create a shortcut in the start menu. When you use the SETUP installation method follow the instructions of the setup. You cannot relocate the application's folder when you use this method without uninstalling and reinstalling Ascolog Insight. 10/147 The SETUP installation method is provided as self extracting archive and as ZIP archive. Self extracting archive ZIP Archive Click on the self extracting archive to start the Extract the ZIP archive to a folder of your choice installation. and click on the file "setup.exe" to start the Just one single file. Contains the files "setup.exe" and installation. "AscologInsigh.msi" Does not work with Avast Antivirus Software, read more on www.ascolog.com/node/37 on Download when you want to access the Windows Installer file ("AscologInsight.msi"). our website. The executable files will be installed in the folder that you specify during setup, Ascolog Insight will suggest the default program folder of Microsoft Windows® (e.g. C:\Program Files\Ascolog\Ascolog Insight or C:\Program Files (x86)\Ascolog\Ascolog Insight on English versions of Microsoft Windows®). The setup will store the application's data (e.g. layouts) in the folder: “%PROGRAMDATA%\AscologInsight” This folder will be referred to as data folder in this documentation. %PROGRAMDATA% is an environment variable of Microsoft Windows®. Consult a documentation about Microsoft Windows® on how to find out the value of this variable. However, try to look up the folder “C:\ProgramData” first because it is quite common that this variable points to this path. 2.4 How to Uninstall 2.4.1 Uninstalling an XCOPY Installation If you installed Ascolog Insight using XCOPY you just have to delete the folder where Ascolog Insight was copied to during installation. 2.4.2 Uninstalling a Setup Installation If you installed Ascolog Insight using the setup it's the best to use the uninstall feature of Microsoft Windows®. Consult a documentation about your version of Microsoft Windows® to learn more. The setup will not uninstall the application's data folder (e.g. layouts) from the %PROGRAMDATA%\AscologInsight folder. You have to manually delete this folder. 11/147 3 User Interface 3.1 Workspace (!) Figure 3.1: Ascolog Insight user interface The workspace of Ascolog Insight and its structure is shown in Figure 3.1. 12/147 3.2 Menu, Toolbar and Context Menus Using the Menu or the Toolbar you can access the commands of Ascolog Insight. The Menu and the Toolbar display what key combinations (shortcuts) can also be used to quickly select a command. In the View menu you can find the Toolbar command which you can use to display or hide the Toolbar. Many graphical elements (e.g. Log window, Functions tree, etc.) have context menus which contain commands related to the data displayed by these graphical elements. Context menus are a very fast and convenient way of working with Ascolog Insight. Right-click on a graphical element to open its context menu. 3.3 Windows In order to to display or hide windows open the View menu, go to Windows and select or deselect a window. 3.3.1 Log Window The Log window displays the log records to be analyzed. The Log window consists of tabs which provide different views on the log records. What is shown in a a tab depends on the used analyze functions (filter expressions, etc.). The Log window always consists of a Main tab which is a common view on all opened log files. Other tabs display e.g. log records belonging to a certain file or log records of a certain log type. In the figure there are three tabs: the Main tab displaying all files, a tab showing log records from the file “COS.LOG” and a tab showing log records from the file “SIN.LOG”. All tabs are listed in the Tab list. Select a tab in the Tab list to set the focus to this tab. The Log window displays preprocessed information from log records, e.g. labels are used instead of hexadecimal codes. In the Log window a table is used to display logs records. A row represents a log record. A column contains information of a certain type, e.g. a timestamp, the type of a log record (error, warning, info), etc. Information of a certain type from a specific log record is displayed in a graphical element called cell. Open the View menu and select Gridlines to display lines between the rows and the columns. 3.3.1.1 Configuring Columns In order to change the properties of a column you can either use the context menu or the View menu. Right-click on a column to open the column's context menu. In the context menu you can make the following settings listed in Table 3.1. Command Description Visible When ticked the column is visible in the Log Align Right When ticked the data of the column is right- window. aligned, otherwise it is left-aligned. 13/147 Type Selects the type of the data that the columns contains. You can select String, String case- insensitive, Float or Integer. The data type is important when sorting the data of a column. Example: Sorting a column which only contains numbers is done differently for the data types Integer and String. Color Rule Select... Assigns a color to a column by creating a color rule. See 5.6 Color Rules for more information. You can select a color from the Color Rule dialog. Random Ascolog Insight will randomly create a color for the column. Table 3.1: Column configuration In the Log window you can also change the width of a column by dragging the column separator (the black vertical line between two column headers) to a different position. In order to change the order of columns drag the column headers to different positions. Figure 3.2: Configuring column settings 14/147 You can see all available columns in the View/Columns menu. In this menu you can select a column to make it visible or invisible. A column which is ticked is visible. When you select View/Column Settings... the dialog shown in Figure 3.2 will open. Select a column and use the Arrow keys to move a column to a different position in the Log window. The order of the rows in the Columns Settings dialog reflects the order of the columns in the Log window. If you changed the order of the columns press the Restore button to restore the original order. If the changes are fine press the OK button to save them, press the Cancel button to reject any changes. Press the Reset button to undo the changes made to the previously modified column. Select a column and press the Edit button to change the settings of a column. This will open the Column Settings Edit dialog shown in Figure 3.3. You can basically make the same settings as in the column's context menu but you can additionally change the width, which is given in pixels, of a column. Figure 3.3: Editing column settings Press the OK button to save the changes or press the Cancel button to discard any changes. 3.3.1.2 Log Record Selection In the Log window the currently selected log record is displayed in blue. It is only possible to select the complete content of a cell. A cell is selected when the mouse hovers over it, the background color will change to light gray. In the Details window it is possible to select any part of a log record. You can use the Toggle Selection command from the Views menu to toggle the selection between the selected log record in the Log window and its file in the Files window. 3.3.1.3 Plain Text The Log window displays processed information (e.g. meaningful descriptions instead of hexadecimal codes). However, sometimes it is useful to see the original content of a log file. You 15/147 can achieve this by selecting a log record and by opening the Plain Text dialog. In order to open the Plain Text dialog (see Figure 3.4) open the View menu and select the Plain Text... command or press the Show Plain Text toolbar icon. The log record which is selected in the Log window (blue background color) will also be selected in the Plain Text dialog (orange background color). Another possibility to open the Plain Text dialog is to use the context menu. Right-click on a cell and select Plain Text... from the context menu. The log record which was selected by right-clicking on one of its cells will be marked with an orange background color. Figure 3.4: Plain Text dialog 3.3.2 Functions Window The Functions window contains the Functions tree which displays a node for each analyze function (color rule, record filter, graph, etc.) any other supported element that was created for analyzing logs. Each node of the tree has a check box. If it is checked the element is enabled, otherwise it is disabled. All sub-nodes of an unchecked node will also be disabled even if their check boxes are ticked. Enabling or disabling a node won't change the status of the check boxes of sub-nodes. This is very useful because it enables you to quickly disable and enable all sub-nodes of a node without losing any previously configured check box settings of these sub-nodes. 3.3.3 Details Window By default the Details window displays the complete log record (=column FULLROW) which is currently selected. However, you can configure in the Options dialog at Columns/DisplayColumn that another column should be displayed. If you display the column FULLROW you will see the 16/147 original information as it is logged in the log files, this means that e.g. any hexadecimal codes are not replaced by more readable descriptions. Besides the Details window there is also a Columns window (not shown in the figure) which is like the Details window but for a specific column. 3.3.4 Graph Window More information about the Graph window can be found in chapter 5.10 Graphs. 3.3.5 Files Window The Files window lists all files with their names (column Filename) and size (column Size) which are currently processed by Ascolog Insight (see Figure 3.5). Figure 3.5: Files window The Status column displays the current status of processing of a file. The Record Filter column shows whether a record filter is applied to a file. In such a case only those log records are retrieved that meet the filter criteria. In the figure a record filter is used for the third file (“SampleFile5.log”) in the list so that only log records are retrieved that contain “a problem occurred” in the TEXT column. The Retrieved Records column displays the number of records that are loaded from a file. The Processed Records column displays the number of records that have been processed from a file. The file “SampleFile5.log” contains 56 log records, all of them were processed but only those two log records which contain “a problem occurred” in the TEXT column were retrieved to be displayed in Ascolog Insight. Click on the header of a column to sort the files in the Files window by the values of this column in ascending (an Arrow up icon is displayed in the header) or descending (an Arrow down icon is displayed in the header) order. The way how the files in the Files window are highlighted only depends on the tab which currently has the focus. A file is displayed with an orange background when a log record that originates from this file is currently selected in the tab which has the focus in the Log window. Files whose records will not be displayed in the currently active tab due to a record filter are highlighted in gray instead of black. Bookmarks are a special case. Files whose log records have bookmarks are always highlighted in black For more information about the analyze functions mentioned in Table 3.2 see 5.3 How to create Analyze Functions (!). 17/147 The context menu of the Files window allows you to quickly execute file-related operations. In order to open the context menu select one or more files and right click. The available commands are listed in Table 3.2. Command Description Record Filter Creates one record filter for the selected file(s) by combining the selected file names using a logical OR in the filter expression: e.g. FILE == <selected file name 1> || FILE == <selected file name 2> Color Rule Creates one color rule for the selected file(s) by combining the selected file names using a logical OR in the filter expression: e.g. FILE == <selected file name 1> || FILE == <selected file name 2> Bookmark Creates one bookmark for the selected file(s) by combining the selected file names using a logical OR in the filter expression: e.g. FILE == <selected file name 1> || FILE == <selected file name 2> Graph/Timeline Creates one timeline graph for the selected file(s) by combining the selected file names using a logical OR in the filter expression: e.g. FILE == <selected file name 1> || FILE == <selected file name 2> Tab Linked, Custom Creates one linked or custom tab for the selected file(s) by combining the selected file names using a logical OR in the filter expression: e.g. FILE == <selected file name 1> || FILE == <selected file name 2> Merge Color Rule Changes the merge color rule of the selected file(s). The merge color rule is applied to the log records when they are displayed in the Log window. Select The user selects a color. Random Ascolog Insight randomly selects a color. Color Rule Creates an own color rule for each of the Group selected file(s) with FILE == <selected file name> as filter expression. Bookmark Creates an own bookmark group for each of the 18/147 selected file(s) with FILE == <selected file name> as filter expression. Graph/Timeline Creates an own timeline graph for each of the selected file(s) with FILE ==<selected file name> as filter expression. Tab/Linked, Tab/Custom Creates an own tab (linked or custom) for each of the selected file(s) with FILE ==<selected file name> as filter expression. Plain Text... Displays the original content of the selected file in its own window. This command is only available when exactly one file is selected. Show All Displays log records from all log file(s) in the Log window Show Exclusive Displays only log records of the selected file(s) in the Log window. Please consider the special treatment of bookmarked log records. Hide Hides the log records from the selected file(s) in the Log window and displays all log records from files that were not selected. Please consider the special treatment of bookmarked log records. Retrieve All Retrieves all log record(s) of the selected file(s). The command is only available when there are any log records that can be retrieved and have not been already retrieved. Unload Unloads the selected log file(s), i.e. the log records of the files are no longer available in Ascolog Insight for analysis. Refresh Refreshes the selected log file(s) only when the modification dates of the log file(s) are newer than the date of the last refreshing. Force Refresh Refreshes the selected log file(s) independently of their modification dates. Reset Sort Order This command is only available when the files in the Files window are sorted by the values of a Files window column. Originally the files are sorted by the order they were loaded. Reset Sort Order restores this original sort order. Select All Selects all files which are listed in the Files window. 19/147 Copy Copies the file name(s) of the selected file(s) to the clipboard. Table 3.2: Context menu of the Files window 3.3.6 Clipboard Window There is also a Clipboard window (not shown in Figure 3.1) that displays the content of the clipboard (Copy & Paste). 3.4 Checking the Application's Status The loaded layout is displayed in the Status bar. The Status bar also displays how many log records are displayed and the total number of registered log files. However, very useful information about the status of different features is also displayed in the Toolbar: • Enable Filter/Disable Filter • Reset Zoom • Merge Records/Recalc Records • Enable Scheduled Task/Disable Scheduled Task In the View menu you can find the Status Bar command which you can use to display or hide the Status Bar. 3.5 Sorting Log Records Right click on the column headers in the Log window to sort the log records using the data from this column. If you click on a header an arrow icon will appear. It indicates that the column is used for sorting. If the data is sorted in ascending order an Arrow up icon is shown, if it is sorted in descending order an Arrow down icon is displayed. You can also sort log records by several columns. Open the View menu and select Record Sort Order... to open the dialog shown in Figure 3.6. All columns which are listed in the Sort Order list are used for sorting log records. First the log records are sorted according to the first column in the Sort Order list. The order of the log records determined by the first sort column is not touched when sorting the log records according to the second column, i.e. only log records which have the same value in the first sort column are compared when sorting according to the second sort column. The same proceeding is used for additional sort columns. In order to add additional columns to the Sort Order list select a column in the Columns list and press the Add button to add a column to the end of the sort order or press the Insert button to insert it before the column which is currently selected in the Sort Order list. Use the Switch button to change the sort order of the currently selected sort order column from ascending (arrow up) to descending (arrow down) and vice versa. When you press Clear all columns are removed from the Sort order list. Select a column in the Sort order list and press the Remove button to remove the 20/147 selected column. You can use the Arrow buttons to move a selected column to a different position in the Sort order list. Figure 3.6: Record Sort Order dialog Press the OK button to sort the log records according to the made settings or press the Cancel button to not change the sort order. The sort order settings will be applied to all tabs of the Log window. In the column headers in the Log window you can see black arrows which indicate whether the column is sorted in ascending or descending order. Moreover, these arrows contain a number which indicates the order of the columns in the Sort Order list of the Record Sort Order dialog. 3.6 Mapping Data onto Descriptions and Icons In the Display Mapping dialog you can map data from log records with descriptive texts or icons to improve the readability of these log records in the Log window. In the figure Figure 3.7 a mapping is shown that modifies the column TYPE of the Log window. When in the TYPE column one of the numbers specified in the Display Mapping dialog's Value column occurs the number is replaced by the appropriate description and icon from the Display Mapping dialog's Text respectively Icon column, e.g. when the number “2” occurs in the TYPE column in the Log window it is replaced by “Warning” and the yellow exclamation mark icon. 21/147 When you press the Clear button all mappings will be deleted. Pressing the Remove button only deletes the currently selected mapping. To save any changes in the Display Mapping dialog you must press the OK button. Pressing the Cancel button but will discard any made changes. Figure 3.7: Display Mapping dialog To create a new mapping press the Add button, to edit an existing mapping select the mapping in the table and press the Edit button. These actions will open either the Add Mapping or the Edit Mapping dialog (see Figure 3.8). However, there are not many differences between these two dialogs except that the Edit Mapping dialog displays the existing mapping information whereas the Add Mapping dialog displays no such information. The application only supports icons of type “.ico”. Ascolog Insight is delivered with icons for the log record type. These icons are located in the “..\_SHARED_\Icons\” folder. 22/147 Figure 3.8: Edit Mapping dialog 23/147 4 Working with Logs and Layouts 4.1 An Introduction to Logs and Layouts (!) This is a very crucial section because it explains the basic concepts which are needed to work with Ascolog Insight effectively. Events which occur in a software system are often documented by log records. The format (structure) of log records from various sources (e.g. software applications) can be very different and depends on the following two aspects: • the information that is provided • the way how the information is provided The following examples show two simple logs that have a different format: Example 4.1 and Example 4.2 show log records of two different types. 2011-10-10, 15:01, ERROR, File not found. 2011-10-10, 15:10, WARNING, Screen saver is turned on. Example 4.1: First example for a log record type The log records in the Example 4.2 provide more information than the records in Example 4.1 because they also contain a severity information (“High” or “Low”). Moreover, the logs use a different way to display the timestamp and to separate the different pieces of information (comma vs. space). 10.10.2011 15:01:12 High Warning <File error> 10.10.2011 15:10:13 Low Information <User A not found> Example 4.2: Second example for a log record type In order to analyze logs Ascolog Insight must identify the pieces of information which are provided by a log record. A log record usually consists of two parts (see Example 4.3): • a header which provides recurring meta information about the logged event (e.g. severity, time of occurrence, etc.) • a description of the event Header 10.10.2011 15:01:12 High Warning 10.10.2011 15:01:13 High Info User E, User F, User G!> 10.10.2011 15:01:14 High Error Description <Access denied!> <Created User A, User B, User C, User D, <File not found!> Example 4.3: Second example for a log record type In order to identify the different pieces of information that a log record provides Ascolog Insight uses a formal grammar (often simply called grammar). The application handles all types of log formats that can be described by a context-free grammar of type LL1. Further log formats can be 24/147 handled by using custom columns definitions which are explained in chapter 10 Custom Columns Definitions (CCD). In most cases the grammar (as part of a layout) which describes your log records already exists. If the needed grammar does not exist you can use Ascolog Insight's layout creation wizard to automatically create a grammar for your purposes. This means that most users can effectively analyze their logs without having a knowledge of formal grammars. If you need support when creating a grammar don't hesitate to contact us. We will be glad to help you. Since there are many good introductions to formal grammars in the world wide web this documentation won't provide an introduction to grammars but focuses on how grammars are used in Ascolog Insight (see chapter 13 Grammars -Advanced Topics). A good introduction to formal grammars can be found in the article Formal Grammars in the English Wikipedia (http://en.wikipedia.org/wiki/Formal_grammar). As stated already above a grammar is part of a layout and describes the format of a log (e.g. the log format used by your software application or the log format used by a certain web server). However, to be more precisely, a grammar only describes the header of a log record. A header must be located at the beginning of a line. This is useful because so it's possible to log data that has the same structure as the header like in Example 4.4 where the header is just a timestamp (text in black) and the description of the event (red text) contains a date in the same format as the header's timestamp: 2014-04-05 13:14:12,123 The time of the last registration was: 2014-04-05 13:12:13,123 Example 4.4: Header and description contain a timestamp of the same format However, if the header is known the description is also defined. The description is all data between two headers (or the end of the log for the last log record in a log file). Log records that span several lines are supported but the header part cannot span several lines. The pieces of information that are identified by a grammar are organized in columns. For the description the column TEXT is used. A layout stores all information which relates to working with a certain log type. It consists of the following main components: • Grammar definitions • Custom columns definition • Configuration settings Besides columns created by the nonterminal symbols of a grammar definition Ascolog Insight also knows custom columns which are generated by the commands of a custom columns definition. In order to get an idea of a custom columns definition have a look at the extracts shown in Example 4.5 and Example 4.6. It is not necessary to understand the command's syntax in detail. 25/147 In Example 4.5 the command map will allow proper sorting by mapping different type strings to values, e.g. if the column _TYPE contains the string "T" or "Trace", then the column TYPE will contain the string “0”. TYPE = map(_TYPE, _TYPE, "T(race)?", "0", "I(nfo)?", "1", "W(arning)?", "2", "E(error)?", "3"); Example 4.5: How the map command can be used in a custom columns definition In Example 4.6 the command xtr will extract the memory usage expressed as a percentage from the FULLROW column and create a new column called MEMORY for the extracted values. MEMORY = xtr(FULLROW, "Memory:[ %09]+", "-?[0-9]+", "\\%"); Example 4.6: How the xtr command can be used in a custom columns definition The configuration settings mentioned in the enumeration above store information like paths to files, macro defines or pattern definitions. Usually the layout you need already exists and just has to be selected by using the Load... command from the Layout menu. How to create a layout yourself is described in 4.3 Creating a Layout with the Wizard (!) respectively 4.4 Creating a Layout without the Wizard. There is also the possibility to purchase customized layouts from Ascolog. In Ascolog Insight a table is used to display logs records. A row represents a log record. A column contains information of a certain type, e.g. a timestamp, the type of a log record (Error, Warning, Information), etc. 4.2 Working with Existing Layouts (!) To start working with logs the first step is to load the appropriate layout if it is already existing. Ascolog Insight is delivered with some default layouts. In order to open a layout go to the Layout menu and you select Load... or Reload... The command Reload… does the same as Load... with the exception that any open log files are not closed. Both commands will open a file browser dialog (see Figure 4.1). Browse to the location where the layout is stored. The default location for the layouts in the file system depends on the installation method you used (setup or XCOPY). Where you find the default location is described in 2.3.1 Installing with XCOPY (!) respectively 2.3.2 Installing with the Setup (!). Select the layout file you want to open and press Open. Layout files have the extension “.xml”. In the Status line you see the currently used layout. 26/147 Figure 4.1: Loading a layout 4.2.1 Saving a Layout (!) The Layout menu offers the possibility to save layouts (Save or Save as...). When you use Save as... you have to specify a file name for the layout. 4.2.2 Editing a Layout A layout consists of several files. Open the Layout menu and select Edit Layout to change any of these files or to enable service support (see chapter 15 Ascolog Insight as a Windows® service. This will open the dialog shown in Figure 4.2. If the settings are correct press Save to change the layout or Save & Reload to change the layout and to immediately apply the changes. 27/147 Figure 4.2: Editing a layout 4.2.3 Sample Layouts Delivered with Ascolog Insight (!) Ascolog Insight provides sample layouts and the appropriate log files for some of these sample layouts within an Ascolog Insight installation. The layouts are located in the “LAYOUT” folder and the log files can be found in the “SAMPLES/LogFiles” folder which are both located in Ascolog Insight's data folder. The log files which belong to a layout are located in a folder which has the same name as the layout's folder. Where you can find the data folder depends on the used installation method and is described in 2.3.1 Installing with XCOPY (!) respectively 2.3.2 Installing with the Setup (!). Some layouts contain filter expressions which depend on file paths to log files. In order to use these layouts without any modifications the log files have to be located in this folder: “C:\ProgramData\Ascolog Insight\LogFiles”. 4.2.3.1 Plain Text Use this layout for any log format where no specific layout is available. It's purpose is that you can view logs of any type. In this layout a line in a log file is considered to be a log record. The layout does not identify a header and thus each line from the log is displayed in a cell of the TEXT column. This means that log records which span several lines cannot be recognized with this layout. Since no columns besides the TEXT column are used the analysis of log records with this layout is limited. 4.2.3.2 Layouts used in the Video Tutorials The layouts created in the video tutorials are TestLayout.xml, TestCSVLayout.xml and TestSampleTableLayout.xml. 28/147 4.2.3.3 Apache Logging Services™ The Apache Logging Services™ project maintains open-source software related to the logging of application behavior. More information can be found at http://logging.apache.org/. The Apache Logging Services™ project consists of several logging frameworks like log4j™ for Java, log4net™ for Microsoft® .NET™, log4Cxx™ for C++ and log4php™ for PHP. These frameworks offer the possibility to configure the format of the log records which are created by them. You can even create log formats that use XML tags to markup the pieces of information of a log record. You can use Ascolog Insight to analyze log files created by these frameworks. You just have to use a layout with the appropriate grammar and custom columns definition. The creation of a layout that describes log record formats of the Apache Logging Services™ without XML is quite straightforward and described in 4.3 Creating a Layout with the Wizard (!) or 4.4 Creating a Layout without the Wizard. For more information about XML based log formats please have a look at 4.2.3.4 XML based log formats. 4.2.3.4 XML based log formats Ascolog Insight is able to work with log formats that use XML tags to markup the information of a log record. Please consider that XML tags increase the size of the log files and that it's hard to read a log file with XML tags without an appropriate tool. The advantage is that XML tags facilitate the processing of such log files with other software tools. However, it's also possible to use a log format without XML and to use Ascolog Insight to convert such a format to a format that contains XML tags (see chapter 14 Exporting Logs, section 14.3 XML). <log4j:event logger="XML" timestamp="1355320268501" level="ERROR" thread="9"> <log4j:message>File not found</log4j:message> <log4j:locationInfo class="Logging.Logger" method="LogError" file="C:\data\Logger.java" line="50"/> </log4j:event> <log4j:event logger="XML" timestamp="1355320269284" level="INFO" thread="9"> <log4j:message>Entering Method</log4j:message> <log4j:locationInfo class="Logging.Logger" method="CreateFile" file="C:\data\Logging\CreateFile.java" line="38" /> </log4j:event> Example 4.7: A simple log4j log record based on XML The layout log4j which is located in the folder “log4jXML” delivered with Ascolog Insight is a sample layout which demonstrates how to create layouts that can work with log formats based on XML tags. The XML log files are created with the log4j™ logging framework. The sample layout is able to process log records as listed in Example 4.7. The used timestamps are based on the UNIX time. The purpose of the log4j sample layout is to be used as a template for other layouts that have to deal with XML based logs. The approach is to use a grammar definition which just identifies the first tag of a log record, in the log4j sample it's the tag <log4j:event. The complete grammar is shown in Figure 4.3. Please note that the terminal in the grammar does not specify the closing angle bracket of the log4j:event tag because the tag has attributes (logger, timestamp, level). 29/147 Figure 4.3: Sample grammar definition for log record based on XML The grammar is used to define what is a log record. So far nothing is different to layouts for log formats without XML. However, in order to extract all the other pieces of information marked by XML tags a custom columns definition is used. More information can be found in section 10 Custom Columns Definitions (CCD).You can find the commands which extract the information in the file “log4jXML_basecolumns.cdf” which is also located in the folder of the log4j layout. Example 4.8 shows the “log4jXML_basecolumns.cdf” file. 30/147 #pragma once #include <basemacros.cdf> #define FMT(text) rpl(text, "<br/>", "%0d%0a", "&lt;", "<", "&gt;", ">", "&amp;", "&") LOGGER = xtr(TEXT, "logger=\"", "[^\"]*", ""); TIMESTAMP = cdt(dat($tmpUTC, xtc(TEXT, "timestamp=\"", "[^\"]*", "")), TIME_UTC2TS_MS(dat($tmpUTC))); LEVEL = map(xtc(TEXT, "level=\"", "[^\"]*", ""), null, "DEBUG", "0", "INFO", "1", "WARN", "2", "ERROR", "3", "FATAL", "4"); THREAD = xtc(TEXT, "thread=\"", "[^\"]*", ""); #define PROF_TID_COLUMN #define FMT_ISO_TIMESTAMP_COLUMN THREAD 0 #include <extendedcolumns.cdf> MESSAGE = chk( xtr(TEXT, "<log4j:message><!\\[CDATA\\[", "", "\\]\\]>"), FMT(xtr(TEXT, "<log4j:message>", "[^<]*", ""))); CALLSTACK = chk( xtr(TEXT, "<log4j:throwable><!\\[CDATA\\[", "", "\\]\\]>"), FMT(xtr(TEXT, "<log4j:throwable>", "[^<]*", ""))); CLASS = xtr(TEXT, "locationInfo class=\"", "[^\"]*", ""); METHOD = FMT(xtc(TEXT, "method=\"", "[^\"]*", "")); SOURCEFILE = xtc(TEXT, "file=\"", "[^\"]*", ""); LINE = xtc(TEXT, "line=\"", "[^\"]*", ""); Example 4.8: A custom columns definition is used to extract the logged information marked by XML tags 4.2.3.5 Microsoft Windows® Event Logs Microsoft Windows® and applications running on Microsoft Windows® can send logs to the Microsoft Windows® event log. There is an application called Event Viewer that is part of Microsoft Windows® which can be used to view these event logs. Consult a documentation about your version of Microsoft Windows® to learn more. 31/147 Figure 4.4: Scheduled Tasks You can adapt Ascolog Insight to analyze these event logs. This is demonstrated by the eventlog layout. In order to access event logs it is necessary to schedule the Search and Refresh tasks AND to enable task scheduling (see chapter 9 Task Scheduling). By default these settings will be correctly set by the eventlog layout. Figure 4.4 shows the required settings in the Scheduled Tasks dialog. The paths in the Schedule Tasks dialog are relative to the folder where Ascolog Insight stores its data. The location of the data folder depends on the installation method you used for installing Ascolog Insight. How to find the data folder is described in 2.3.1 Installing with XCOPY (!) respectively 2.3.2 Installing with the Setup (!). In the “eventlog.ini” file that is located in the folder “PLUGIN_CFG”, which is a sub-folder of the data folder, you can configure what categories of event logs shall be displayed. You can also set a limit for the number of log records that shall be retrieved. Do not confuse the “eventlog.ini” file of the eventlog layout which is located in the layout's folder with the “eventlog.ini” file from the “PLUGIN_CFG” folder. The semi-colon in the “eventlog.ini” file in Example 4.9 marks a line as a comment, thus in the example below only log records logged from the Application category are retrieved. You can switch to log records of the category System by removing the semi-colon from the “System line” and adding it to the “Application line”. The default “eventlog.ini” file from the “PLUGIN_CFG” folder is listed in Example 4.9. 32/147 [EVENTLOG] sourcename = Application ; sourcename = System max_records = 1000 Example 4.9: The configuration file "Eventlog.ini" The eventlog layout has some predefined graphs. You can find these graphs in the Functions tree. There is also a layout file called “eventlog2syslog_service.xml”. This layout works together with the syslog layout and is required in the syslog sample. For more information please refer to chapter 14.6 Syslog. 4.2.3.6 Windows® Installer (MSI) Windows® Installer can be configured to create logs about the installation process. These logs can be used to analyze installation problems. Ascolog Insight provides the MSI layout which describes the log format used by the Windows® Installer. Consult a documentation about your version of Microsoft Windows® to learn more. A good place to look for Windows® Installer logs is the folder for temporary files of Microsoft Windows®. Using Ascolog Insight you can analyze the log files created by the Windows® Installer. Be careful not to confuse these log files with the well-known installation packages. The installation packages are commonly called MSI files because of their file extension. 4.2.3.7 OutputDebugString The OutputDebugString function belongs to the API of Microsoft Windows® and sends a string to the debugger for display. Consult a documentation about Microsoft Windows® to learn more. You can adapt Ascolog Insight to analyze the messages created by the OutputDebugString function. This is demonstrated by the OutputDebugString layout. In order to access these messages it is necessary to schedule the Search and Refresh task AND to enable task scheduling (see chapter 9 Task Scheduling). By default these settings will be made by the OutputDebugString layout. Figure 4.5 shows the required settings of the Scheduled Tasks dialog. The paths in the Schedule Tasks dialog are relative to the folder where Ascolog Insight stores its data. The location of the data folder depends on the installation method you used for installing Ascolog Insight. How to find the data folder is described in 2.3.1 Installing with XCOPY (!) respectively 2.3.2 Installing with the Setup (!). 33/147 Figure 4.5: Scheduled Tasks dialog In the “OutputDebugString.ini” file that is located in the “PLUGIN_CFG” folder (see Example 4.10) you can configure a limit for the number of messages that shall be retrieved from the OutputDebugString log. Do not confuse the “OutputDebugString.ini” file of the layout which is located in the “OutputDebugString” layout folder with the “OutputDebugString.ini file from the “PLUGIN_CFG” folder. [ODS] max_records = 100000 Example 4.10: The default “OutputDebugString.ini” from the "PLUGIN_CFG" folder 4.2.3.8 Syslog Ascolog Insight is delivered with a syslog sample layout that is based on Ascolog Insight's syslog plug-in. The sample demonstrates how syslog messages are sent from a syslog client to a syslog server. The syslog sample layout is described in 14.6 Syslog. 34/147 4.2.3.9 Layouts for HTML and HTML Report Ascolog Insight is delivered with a sample plug-in called HTML that converts logs to HTML. An HTML table is used to display the logs. The HTML plug-in is only a demo. The HTML Report plug-in is another sample plug-in and very similar to the HTML plug-in. It generates an HTML file which consists of the Ascolog logo at the top , followed by the active Graph windows and at the bottom the log records are displayed in an HTML table in the same way as in the HTML plug-in. The layout TestLayout (file “TestLayout.xml”) shows how to use the HTML and HTML Report sample plug-ins. The plug-ins are described in chapter 14.4 HTML and 14.5 HTML Report. 4.2.3.10 Timestamp There a two different timestamp layouts. Their XML files have the following names: • “Timestamp.xml” • “Timestamp_extracted.xml” Both layouts use regular expressions to identify a wide range of timestamps. The “Timestamp.xml” layout uses a grammar to identify timestamps whereas the “Timestamp_extracted.xml” layout identifies the timestamps in the custom columns definition. The “Timestamp.xml” layout only identifies timestamps which are located at the beginning of a log record. The “Timestamp_extracted.xml” layout can identify timestamps at any position in a log record. The identified timestamps will be displayed in their own column. Only the first timestamp of a line is identified and considered to be part of the header. If there are more timestamps in the line they belong to the log's description. The timestamps should have the same order for year, month and day in the date. Otherwise the ISO_TIMESTAMP column might be inconsistent (see 10 Custom Columns Definitions (CCD), ISO_TIMESTAMP column and option SourceTimestampFormat). Both layouts identify timestamps of the following formats: The date can start with the year or with the day and the year can consist of two or four digits. Month and day must consist of two digits. Year, month and day must be separated either by slashes or dots or hyphens. Time and date must be separated by a single white space. Hour, minute and second must consist of two digits which are separated by a colon. Milliseconds are optional. If they exist they must consist of three digits and they must be separated by a dot or a comma from the seconds. Some examples of supported timestamp formats are shown in Example 4.11. Notation: Y=Year, M=Month, D=Day, h=hour, m=minute, s=second, m=millisecond YYYY-MM-DD, hh:mm:ss,mmm YY/MM/DD, hh:mm:ss.mmm DD.MM.YY, hh:mm:ss DD-MM-YYYY, hh:mm:ss Example 4.11: Some supported timestamp formats 35/147 Open both the ISO_TIMESTAMP and the TIMESTAMP column in the Log window and display them next to each other. So you can easily compare the format of the original timestamps from the logs and the ISO timestamp format. 4.3 Creating a Layout with the Wizard (!) If the layout does not already exist you can easily create a layout with the layout creation wizard. This chapter will demonstrate how to create a layout for the log records shown in Example 4.12 with the layout creation wizard. 2013-02-27 22:56:37,206 Info Low User A logged in. 2013-02-27 22:56:37,206 Error High File not found! Example 4.12: A layout will be created for these log records Select a log record in the Log Window, open the Layout menu and select Creation Wizard... The dialog shown in Figure 4.6 will open and the selected log record will be displayed in the Header selection edit control. If you do not select a log record and start the layout creation wizard you won't see this dialog but the one shown in Example 4.7. In all dialogs of the wizard you can use the Back and Next buttons to switch between the dialogs of the wizard. Press the Cancel button to stop the layout creation wizard without creating a layout. Figure 4.6: Header selection As it is shown in Figure 4.6 select the header of the log record. A header provides meta information about the logged event (e.g. severity, type, time of occurrence, etc.) and it is 36/147 recurring. If you are not sure what you should select read the section 4.1 An Introduction to Logs and Layouts (!) to get an idea of log headers. In Figure 4.6 the header is already selected which is indicated by the blue background color. The header consists of the timestamp, the log record type (“Info”), the log level (“Low”) and the separators between the header and the description which is in this example a colon and a white space. It is generally a good idea to include the header/description separators in the header so that they do not indent the description in the TEXT column. The text “User A logged in.” is not part of the header but of the description because it described the logged event. After you have selected the header press the Next button to proceed. The header is analyzed by the wizard. It will try to identify the different parts of the header and it will assign columns to the different parts. The columns will be displayed in the next dialog (see Figure 4.6). Figure 4.7: Identified columns of the sample header In the Figure 4.7 the Sample header data edit control displays what was selected in the Header selection of the previous dialog. If you did not select a log record before opening the layout creation wizard of Figure 4.7 is the first dialog you will see and you have to enter the header of your log record in the Sample header data edit control. After you entered or modified the header press the Scan button so that the wizard will analyze the header and create columns for the different parts of the header. The created columns are displayed in the Extracted columns preview. If the columns are correct press Next to continue, otherwise modify the header. You can change the column names in the Grammar Definition dialog after completing the wizard. In the next step the layout creation wizard proposes a name for the layout and a folder for storing the layout. (see Figure 4.8) Ascolog Insight has a default folder for storing layouts which is the proposed by the wizard. The default folder for layouts is the folder “LAYOUTS” which is a sub- folder of the data folder. The location of the data folder depends on the installation method that 37/147 was used for installing Ascolog Insight. How to find the data folder is described in 2.3.1 Installing with XCOPY (!) respectively 2.3.2 Installing with the Setup (!). Press the “...” button to open a file dialog to select a different path. If you check the Create Directory for layout check box the wizard will create an own folder for the layout which is recommended. Press the Next button to continue. Figure 4.8: Layout Store The last dialog shown in Figure 4.9 is a summary so you can verify if the settings you made are correct. If you check the Apply layout check box Ascolog Insight will immediately use the layout after pressing the Finish button, otherwise the currently loaded layout will be used. If the settings are correct press the Finish button to create the layout. If you press the Cancel button the layout creation will be aborted and all the entered information is canceled. 38/147 Figure 4.9: Summary Detailed information about layouts can be found in chapter 12 Layouts – Advanced Topics. When you ticked the Apply layout check box Ascolog Insight will reload the new layout created by the layout application wizard. Reloading means that the layout is loaded and any loaded log files are kept open (see Load and Reload commands described at the beginning of chapter 4.2 Working with Existing Layouts (!)). If you changed the layout that was loaded when starting the layout creation wizard the dialog shown in Figure 4.10 offers you the possibility to save or discard these changes. Figure 4.10: Apply layout check box 4.4 Creating a Layout without the Wizard 4.4.1 Creating the Files of the Layout To create a layout without the wizard go to the Layout menu and select Create... The dialog which is shown in Figure 4.11 will be opened where you have to specify the name of the new layout and the folder that should contain the new layout. The file name should also INCLUDE the file extension “.xml”, i.e. a correct layout name would be “Test.xml”. Since a layout consists of several files it is recommended to create an own folder for your layout. The default location for layouts is the “LAYOUTS” folder which is a sub-folder of the data folder. How to find the default location for the data folder is described in 2.3.1 Installing with XCOPY (!) respectively 2.3.2 Installing with the Setup (!). 39/147 Figure 4.11: Providing the layout file in the layout folder After you have specified the folder and the file name for the layout press the Save button. Another dialog which is shown in Figure 4.12 will be displayed that shows the settings made so far. The Layout file name field shows the destination folder together with the main file of the layout. The other fields display the names of the other layout files. The “...” buttons offer you the possibility to change the file paths when the Default check boxes are unchecked. If the Default check boxes are checked the files are created at the default locations. Another check box of this dialog is Enable service support. Check this box if you want to run Ascolog Insight as a service. If you use Ascolog Insight as a GUI application do not tick this check box. Purchasing an Ascolog Insight Professional license does not entitle you to use the Ascolog Insight SDK or to run Ascolog Insight as a Windows® service. If you are interested in this functionality please contact us to learn more or to get a quote. If all settings are correct press the Save button to create and save the layout. Save & Reload does the same as Save but it additionally also activates the layout so that it is applied to the currently loaded log files. If you press Cancel the layout will not be created and any settings made so far will be discarded. 40/147 Figure 4.12: Files of the layout Detailed information about layouts can be found in chapter 12 Layouts – Advanced Topics. The next step is to create a grammar for the layout. 4.4.2 Creating a Grammar – A Short Introduction A good way to create a grammar is to use Ascolog Insight's scanning feature. Select the log header without the description in the Details window, right click and select the Copy command to copy the header to the clipboard. Go to the Tools menu and open the Grammar Definition dialog (see Figure 4.13). Paste the header into the Text sample edit control (press Ctrl+V) and press the Scan button next to the control. Ascolog Insight will scan the log header and create a grammar definition, i.e. it will create terminals, nonterminals and production rules. In order to select the header of the log records in Example 4.13 you must select one of the texts with the light gray background. These texts are the headers of these log records. A header does describe the logged event. Other characteristics of a header are that it is recurring and that it provides meta information of the logged event. The meta information in the example are the time of logging, followed by an abbreviation for the log type (E for Error, W for Warning) and a hexadecimal number which represents a process ID. The rest of the log record is part of the description since it describes the event that caused logging. 2013-02-27 23:11:22,104 E 0x1D50 Memory usage: 66% (a problem occurred) 2013-02-27 23:11:25,104 W 0x1D50 Memory usage: 22% Example 4.13: Selecting the header of a log record If you select the header of the first log record the Grammar Definition dialog will look like in the Figure 4.13. The selected header is displayed in the Text sample field. 41/147 Figure 4.13: Grammar Definition As you can see Ascolog Insight created 4 terminals and 4 production rules in the Nonterminals list. The first production rule says that a header represented by the nonterminal symbol HEADER consists of a nonterminal called NT_1 followed by a terminal symbol space followed by a nonterminal NT_2, followed by a terminal space and another nonterminal NT_3. The pieces of information of the log record and the nonterminals are related as shown in Table 4.1. Nonterminal Information Example NT_1 Timestamp 27.02.13 23:11 NT_2 Log type E (for Error), W (for Warning), other log types are not shown in the example but are possible. NT_3 Process ID 0x1D50 Table 4.1: Information identified by the nonterminals 42/147 It is useful to change the generic names of the nonterminal symbols to meaningful names. Ascolog Insight cannot do this for you because it cannot know the actual meaning of the nonterminal symbols (e.g. that the nonterminal NT_3 represents a process ID). In order to change the name select the nonterminal you want to modify and press the Edit button. A dialog as shown in Figure 4.14 will be displayed: Figure 4.14: Layout Definition Nonterminal In the Name field enter the name TYPE instead of NT_2. You can leave the other settings untouched. Press the OK button to save the changes or the Cancel button to keep the old settings. In the same way you can change the other names. If you are finished open the File menu of the dialog and press Save & Apply to save the new grammar definition and to apply it to the currently loaded logs. Ascolog Insight uses color highlighting to assign the parts of a text sample to the nonterminals and terminals of a grammar so that you can easily see how the grammar and the text sample relate. The assignment works in both directions: from a text sample to the grammar and from the grammar to the text sample. If no matches can be found nothing is highlighted. You can see some examples in Figure 4.13. This was just a quick introduction about grammars in Ascolog Insight but maybe it is sufficient for your purpose. However, for more details please refer to chapter 13 Grammars -Advanced Topics. 4.5 Loading, Merging, Browsing and Closing Logs (!) Ascolog Insight supports the merging of logs originating from different sources (log files). All logs are displayed in a single scrollable view (Log window) without the need to switch between pages to see a certain log record. Log records from different sources can be easily distinguished by their color in the Log window. The user can configure a default column which is used to determine the order of the log records from the different sources in the merged log. This column is called merge column. It is useful to choose a column that contains comparable data, e.g. the timestamps of the log records. The merged log is displayed in the Main tab of the Log window. 43/147 In order to configure the merge column open the Tools menu and select Options... to open the Options dialog (see Figure 14.5). Expand the category Columns and select MergeColumn. In the Value drop down list select the column you want to use as merge column. Closely related to the merge column is the timestamp column which is used for timestamp related operations like creating a graph. If you set in the Options dialog the TimestampColumn option to <default> the merge column is used as timestamp column. If the merge column does not contain any timestamp you have to set the TimestampColumn option to a column which contains timestamps. Figure 4.15: MergeColumn and TimestampColumn options There are several ways to load log files. In the File menu you can find the command Load... This will open a file browser and you can select the logs you want to load. Previously loaded logs will be closed. In order to merge previously loaded logs with new logs you can use Merge... which will also open a file browser. Instead of using the Merge... command you can also use Drag & Drop. Drag the log files you want to load from a file browser like the Windows Explorer to the window of Ascolog Insight and drop them there. When you open logs with either Load..., Merge... or Drag & Drop the complete log files are loaded into the computer's memory. This mode is called memorymode. In memory-mode you can use the full functionality of Ascolog Insight. You can also open ZIP files. ZIP files will be extracted and their content will be displayed. 44/147 The command Browse... only loads parts of the selected files into memory. The rest of the files are kept on the hard drive and are loaded when they are accessed. This mode is called browsing mode and is recommended if a file or a group of files are too big to be loaded into memory. The browsing mode has the disadvantage that the analyze functions cannot be used and that the files are only displayed in separate views and that they are not merged in the Main tab. If you want to close a log file select the file in the Files window. Right click to open the context menu and select Unload. 4.5.1 Merging Logs of Different Types It is also possible to display logs belonging to different log types in the Main tab of the Log window. Log records of different types do have different headers. If the grammar definition of the loaded layout does not already describe all the different headers the definitions of these headers have to be merged to the grammar definition. You can proceed in a similar way as described in 4.4.2 Creating a Grammar – A Short Introduction but there is one difference. In order to create the desired grammar definition select a record whose header you want to add. The record will be displayed in the Details window. Select the header (do not select the description) and copy the header to the clipboard. Then go to the Tools menu and open the Grammar Definition dialog. Paste the header into the Text sample edit control and press the Scan button next to the control. Ascolog Insight will scan the log header and create a grammar definition, i.e. it will create terminals and nonterminals and replace the previously shown terminals and nonterminals. However, the original terminals and nonterminals are not gone because they are still stored in the grammar definition file but they are just not displayed in the dialog. In order to add the new definitions to the grammar definition file go to the File menu, select Merge and then File... Ascolog Insight will propose the currently loaded grammar definition file as merge target. You can also select a different grammar definition file if you want. Press OK to add the new definitions to the grammar definition file. The old definitions will be kept. Go again to File, press Save & Apply to save the new grammar definition and to apply it to the currently loaded logs. 4.6 Reset If you want to investigate different log files than the currently loaded ones, open the File menu and select Reset. A reset will trigger the following actions: • All loaded log files will be closed. • If Ascolog Insight is in the browsing mode the application will stop the browsing mode. • The timers for automatic refresh and scheduled tasks will be reset. • Auto refreshing will be terminated. The settings of the layout like the created analyze functions will be kept. 45/147 5 Analyze Functions 5.1 An Introduction to Analyze Functions (!) Ascolog Insight provides several functions for analyzing logs. In the following table you can find all available analyze functions with a short description. The analyze functions are described in more detail in the dedicated sections in this chapter. Analyze functions are created and used in a very similar way so what they have in common is described in 5.3 How to create Analyze Functions (!) and 5.4 Filter Expressions (!). These two sections provide a lot of useful information and so it's highly recommended to read them. A tree structure called Functions tree is used to organize all available analyze functions except tabs. The Functions tree is located in the Functions window. The tabs are organized in the Tab list (see Figure 3.1). Table 5.1 lists the analyze functions which are available. Analyze functions Description Record Filter A record filter controls what log records are displayed in the Log window. All log records that match the filter expression are displayed in the Log window. Color Rule Color rules apply a certain color scheme to log records or to parts of a Bookmark Bookmarks are intended to help the user to quickly find log records log record in order to mark important information. again. Graph Graphs are used to visualize the occurrence of log records or their data. Fixed Samples Fixed samples are special graphs. They are used to mark important Tab Tabs are different views of the log records that should be analyzed. The points in time to facilitate the orientation in a graph displayed log records are selected on the basis of the tab's filter expression. Table 5.1: Analyze functions Categories are used to create groups of related analyze functions. Table 5.2 lists the available categories. Category Description Root node The root node of the Functions tree is a special category node that Category Categories are used to create groups of related analyze functions. contains all other nodes. The root node is always available. Table 5.2: Categories On the one hand a category facilitates keeping track of the used analyze functions and on the hand it facilitates applying filter expressions to analyze functions. If a filter expression is added to 46/147 a category this filter expression is applied to all analyze functions which are sub-nodes of the category by combining the filter expressions of the analyze functions with the category's filter expression using a logical AND (&&). You can add exactly one filter expression node to a category. If you want to combine several filter expressions you have to use logical operator nodes (AND, OR, NOT) within the filter expression Figure 5.1 shows a filter expression directly below a category node. Figure 5.1: A filter expression directly below a category node... The Functions tree shown in Figure 5.1 is equivalent to the Functions tree shown in Figure 5.2. Please note how the filter expressions are combined using a logical AND (&&). Figure 5.2: … is equivalent to these filter expressions. Filter expressions are used to define criteria that log records must meet in order to be processed by an analyze function or a category of analyze functions (i.e. all analyze functions which are nodes of this category). Table 5.3 gives an overview of the available filter expressions. Filter expressions Description Generic filter A generic filter expression is used to define complex criteria. Wildcard A wildcard is a special filter expression. It is used to select all log expression records. If you want to display all occurrences of log records in a graph add the wildcard to the graph node. RANGE A range is a filter expression that is defined by two values. All values between these two values will meet the filter criteria. Timeframe A timeframe is a special RANGE based on timestamps. One of the timestamps is the current local time. Table 5.3: Filter expressions There are some more elements of the Functions tree which are called logical operators. They are listed in Table 5.4. Their purpose is to combine filter expression nodes to form more complex filter expressions. 47/147 Logical operators Description AND You can use this node to combine filter expressions with the logical AND OR You can use this node to combine filter expressions with the logical OR operator. operator. NOT You can use this node to negate a filter expression. Table 5.4: Logical operators 5.2 Working with the Functions Tree (!) Figure 5.3: Function tree 48/147 Figure 5.3 shows a Functions tree that contains all nodes which are available. It is possible to disable and to enable each node in the tree. A disabled node is not considered when analyzing logs. If you disable a node all nodes below this node are also disabled even though their check boxes stay ticked. Ascolog Insight assigns meaningful default names to nodes (e.g. New Color Rule or New Record Filter, etc.) when they are created. Use the mouse to drag a node to a different position in the Functions tree. The node will be added at the end. Right-click on a node you want to manipulate to open its context menu. Commands are only available when it makes sense to execute them, otherwise they are displayed in gray. Table 5.5 lists and describes the available commands. Command Description Insert Inserts a new node below the selected node. You can insert analyze functions, categories, filter expressions and logical operators. Parent Inserts a new node above the selected node. You can insert categories, Edit... Opens the Filter Expression dialog where you can edit a filter expression. Rename You can specify a different name for the node. Collapse Filter expressions can be combined using logical operator nodes (AND, logical operators and the filter expression RANGE. OR, NOT) but logical operators can also occur within filter expressions using the following symbols: &&, ||, ! This command transforms filter expression combined by logical operator nodes into a single filter expression with the same meaning. The single filter expression contains all original filter expressions and logical operators. Example: The three nodes (1 logical operator node + 2 filter expressions): OR (TYPE==Error) (TYPE==Warning) are transformed to a single filter expression: (TYPE==ERROR) || (TYPE==Warning) Expand Expand does the opposite of the Collapse command. Ascolog Insight creates a sub-tree consisting of filter expressions and logical operators nodes which form the same Boolean expression as the original single filter expression. The most “smallest” filter expressions are used for the nodes Smallest means that only comparison operators but no &&, || or ! operators are used within a filter expression. Switch Changes an OR node to an AND node respectively an AND node to an OR node. Range Changes an AND node to a RANGE node. Delete Deletes a node and its sub-nodes. 49/147 Cut Copies a node to the clipboard and deletes it from the Functions tree. Copy Copies a node to the clipboard. Link You can use this node to create a link to another analyze function or filter expression in order to reuse them. If the linked node is changed the linking node will reflect these changes. Linking is useful when you want to reuse a node in different positions in the Functions tree because you can make all changes in one single place (at the linked node). In order to create a link the following steps are required: • • Select a node and right-click to open the context menu. Select Link. A link will be created and the link is available in the clipboard. • Select the node where the newly created link should be located and right-click to open the context menu. • Select Paste. A Link node will be created as a sub-node of the selected node. Paste Inserts a node from the clipboard as a sub-node of the currently selected node. Table 5.5: Context menu of the Functions tree 5.3 How to create Analyze Functions (!) There are several ways to create analyze functions or filter expressions for analyze functions. They are described in Table 5.6. Method Description Edit menu (1) Open the Edit menu and select one of the commands: New Node/Record Filter, New Node/Color Rule, New Node/Bookmark or New Node/Graph. This will create an analyze function without a filter expression. You still have to add the filter expression. Edit menu (2) Select the log record which should be bookmarked. Open the Edit menu and select Bookmark/Toggle. This will create a new bookmark for the selected log record. Edit menu (3) Open the Edit menu and select Tab/New... or Add Fixed Sample... A dialog will be opened where you can configure the new tab respectively the fixed sample. Context menu of the Select the log record whose data should be used for creating an analyze Details window function. By default the complete log record (i.e. the column FULLROW) will be displayed in the Details window (this can be changed in the Options). Select the part of the log record that should be used in the filter expression (e.g. “a problem occurred” from the log record's description). Then right- click on the Details window to open the context menu to select an analyze function. An analyze function will be created whose filter expression 50/147 specifies that the column displayed in the Details window must contain the selected text from the Details window: e.g. FULLROW~=”a problem occurred” Context menu of the Right-click on a cell in the Log window or on a file in the Files window to Log window or Files create an analyze function based on the cell's data (e.g. Warning) or on the window file. This will open the context menu where you can select one of the analyze functions: Record Filter, Color Rule, Bookmark, Graph or Tab. Log window: The filter expression of the new analyze function will compare the column (i.e. each cell in this column) where the selected cell is located with the content of the selected cell: e.g. LOGTYPE==Warning where LOGTYPE is a column Files window: The filter expression of the new analyze function will compare the FILE column with the file name of the selected file, e.g. FILE==”MyFile1.log” When more than one file is selected the filter expression of the new analyze function will compare the FILE column with the file names of all selected files, e.g. (FILE==”MyFile1.log”) || (FILE==”MyFile2.log”) Context menu of the All log records which have the same data value in a column are considered Log window or Files to be a group. Right-click on a column and select Group/Color Rule, window – Group Group/Bookmark, Group/Graph or Group/Tab to create analyze functions command based on groups of this column. For each group Ascolog Insight will create its own analyze function whose expression filter is based on the common data value of the group. When you select files in the File window and one of the group commands from the File window 's context menu, an analyze function will be created for each selected file, i.e. in this case a group is formed by all log records that have the same file name in the FILE column. Example for the Log window: If there are the three data values Info, Warning and Error in the LOGTYPE column Ascolog Insight will create three analyze functions with the following filter expressions: LOGTYPE==Info, LOGTYPE==Warning and LOGTYPE==Error Drag & Drop from Hover the mouse over a cell whose data you want to use, left click and drag the Functions tree analyze function or a category. This will create a new filter expression node the Log window to the cell to the Functions tree. Drop it to a node which is representing an below the selected node where the log records should equal or contain the dragged data value in the selected column: LOGTYPE==Info or TEXT*=”User could not be deleted” Drag & Drop from Hover the mouse over a cell whose data you want to use, left click and drag the Log window to the cell to the Tab list. This will create a new tab with a filter expression the Tab list based on the cell's data: SEVERITY==High Tab list Right click on the Tab list to open the context menu. Select New... to create a new tab. The Tab settings dialog will open where you can specify the 51/147 settings for the tab to be created. Keystroke combination You can also use the keystroke combinations which are assigned to the analyze functions. You can check out the assigned keystroke combinations in the menu. Table 5.6: Available methods of creating analyze functions The methods Dragging & dropping respectively using the context menus are very efficient ways to create an analyze function. Hint: You can negate a filter expression when you press the CTRL key while creating the filter expression. 5.4 Filter Expressions (!) Filter expressions specify what log records should be processed by an analyze function or a category of analyze functions. A filter expression has its own node in the Functions tree. Filter expressions can be sub-nodes of analyze functions or of categories. In order to assign a filter expression to an analyze function or a category you have to make it a sub-node of the analyze function respectively of the category. If a filter expression is a sub-node of a category it is applied to all analyze functions that belong to this category. You might think that filter expressions are only used by records filters but they are used by all analyze functions and not only by record filters, e.g. graphs use filter expressions to determine what log records should be displayed. Filter expressions are Boolean expressions, i.e. the expression is either true or false. If Ascolog Insight has to apply an analyze function it evaluates for each log record if the filter expression is true or false. If it is false the log record is ignored (filtered out), otherwise the log record is processed. When you press the CTRL key while creating a filter expression a logical NOT will be applied to the filter expression. 5.4.1 Wildcards The most basic filter expression is a wildcard. It can be used to select all log records. In example shown in Figure 5.4 the wildcard is used to assign a color to all cells of the TIMESTAMP column. 52/147 5.4.2 Timeframe The timeframe is a special filter expression to specify a time period ranging from the current local time to some point in time in the past (see Figure 5.5). What is used as the current local time depends on the Automatic update check box. If the Automatic update check box is not ticked the current local time is determined only once when the OK button is pressed. Figure 5.5: Timeframe configuration When the check box Automatic update is ticked the current local time is determined for the first time when the OK button is pressed but the time is updated when the analyze functions are reapplied, e.g. if you want to display only the log records of the last 10 minutes you have to add the timeframe to a record filter, tick the Automatic update check box in the Timeframe dialog and configure Ascolog Insight to automatically refresh the Log window. The starting time of the timeframe is relative to the current local time and can be configured in the Current local time minus field. You can also configure the time unit that shall be used. A timeframe node used as filter expression of a record filter in the Functions tree is shown in Figure 5.6. Figure 5.6: Record filter which uses a timeframe filter expression The filter expressions timeframe and RANGE can also be added directly under the ROOT node (as shown for the RANGE node in Figure 5.3). In this case all log records which do not match the filter expression are not hidden but displayed with a white background. 5.4.3 RANGE A very common use case is to select log records whose data values are between two specified values (e.g. timestamps must be between 2013-10-25 and 2013-10-27). Since such a filter expression is very often used there is an own filter expression called RANGE for this. By default a range also includes the starting and ending values. In order to use a range the values must be comparable (e.g. numbers, timestamps). A range can be created very easily by using drag & drop. First you create a RANGE node in the Functions tree. The second step is to select a value in the Log 53/147 window and to drag it to the RANGE node. You have to do this again for the second value. In Figure 5.7 you can see a RANGE that uses timestamps. 5.4.4 Generic Filter Expressions In order to define more complex filter expressions Ascolog Insight provides a Filter Expression dialog consisting of the two tabs Components and Occurrence to support the user in defining Boolean filter expressions. 5.4.5 Components Tab Figure 5.8: Filter expression dialog - Components tab Figure 5.8 shows the Components tab. The Selected components list on the right and the Command drop down list on the bottom display the created filter expression. Open the Command 54/147 drop down list to reuse previously created filter expressions. You can either directly enter a filter expression in the Command field or you can create an expression by using the buttons next to the Columns and Selected components lists. Log records consist of different pieces of information which are organized in columns, e.g. timestamps, types of a log record (error, warning, info), etc. Select the column whose information should be evaluated and press the Add button to add a filter expression component to the end of the filter expression or press the Insert button to insert it before the currently selected line in the Selected components list. The Filter Criterion dialog (see Figure 5.9) will pop up where you have to specify an operator and a value that should be compared with the content of the column's cells. Figure 5.9: Specifying a filter criterion for a filter expression component Note that in the dialog above a mapped value is compared (see 3.6 Mapping Data onto Descriptions and Icons). Therefore no double quotes are used in the Values edit control. If a string has to be compared it must be encompassed in double quotes. Use the buttons next to the Selected components list to edit, remove or move (Arrow buttons) the selected filter expression component. Press the Clear button to remove all filter components from the Selected components list. The Components tab also shows you if a filter expression is valid at the bottom of the dialog. Press OK to create the filter expression or Cancel to discard any settings. 5.4.6 Occurrence Tab The Occurrence tab (see Figure 5.10) offers another faster way than the Components tab to specify a filter expression. However, the drawback is that it has some restrictions compared with the Components tab. Log records consist of different pieces of information which are organized in columns, e.g. timestamps, types of a log record (error, warning, info), etc. Select the column from the Column drop down field to specify what column should be compared with the value in the Text field. In the check box next to the Column you can specify whether the column shall or shall not contain the value of the Text field. Use the Search options to refine the filter expression. The Combine options specify how the active filter expression and the newly specified filter expression should be combined. The Components tab shows the active filter expression in the Selected components and 55/147 the Command field. In Figure 5.8 the active filter expression is (TYPE == Warning || TYPE == Error). The newly specified filter is TEXT ?= "a problem occurred" as shown in Figure 5.10. Because the logical AND combination is selected in the Combine options the logical operator && is inserted between these two filter expressions (see Figure 5.10, Command field). The Command field shows Figure 5.10: Filter expression dialog - Occurrence tab the created filter expression. Below the Command field the dialog shows if the created filter expression is valid. Press OK to create the filter expression or Cancel to discard it. 5.4.7 Operators in Filter Expressions Table 5.7 lists all operators which can be used in filter expressions. Operator Description Example == Equal VALUE == 100 <= Less or equal VALUE <= 100 >= Greater or equal VALUE >= 100 56/147 != Not equal VALUE != 100 < Less VALUE < 100 > Greater VALUE > 100 ~= Contains, case-sensitive TEXT ~= "abc" ?= Contains, case-insensitive TEXT ?= "abc" *= Contains, regular expression TEXT *= "(abc)+" || Or TEXT ~= "abc" || TEXT ~= "def" && And TEXT ~= "abc" && TEXT ~= "def" () Group (TEXT ~= "abc" || TEXT ~= "def") && TEXT ~= "ghi" & Escape TEXT ~= "&"" %n String parameter TEXT ~= %1 %nd Integer parameter TEXT < %1d %nf Float parameter TEXT < %1f String value TEXT ~= "abc" Integer value VALUE == -100 Float value VALUE == 100.0 Mapped value TYPE == Error Null value TYPE == NULL Table 5.7: Operators in filter expressions The operator precedence in filter expressions is the same as in the programming languages C++. 5.5 Analyze Functions based on Data Groups (!) This method of creating analyze functions is very efficient which is why it's explained here for a second time (besides 5.3 How to create Analyze Functions (!)). All log records which have the same data value in a column are considered to be a group. Right- click on a column and select Group/Color Rule, Group/Bookmark, Group/Graph or Group/Tab to create analyze functions based on groups of this column. For each group Ascolog Insight will create its own analyze function whose expression filter is based on the common data value of the group. When you select files in the File window and then one of the group commands an analyze function will be created for each selected file, i.e. in this case a group is formed by all log records that have the same file name in the FILE column. Example: If there are the following data values Trace, Info, Warning and Error in the TYPE column and you select Group/Color Rule from the column's context menu Ascolog Insight will create analyze functions with the filter expressions shown in Figure 5.11. 57/147 Figure 5.11: Analyze functions created with the Group command 5.6 Color Rules Color rules apply a certain color scheme to log records or to parts of a log record in order to highlight important information. Since a color rule is an analyze function the usual ways for creating analyze functions can be applied. How to create analyze functions is described in 5.3 How to create Analyze Functions (!) and 5.4 Filter Expressions (!). When you load a new log file Ascolog Insight automatically creates a color rule for this file so that all log records of this file can be recognized by their color. The node of such a color rule is created in the Merge Colors category. When the user creates a color rule the default category for color rules is a category called Color Rules. The exception are color rules for columns created by using the context menu of a column. These color rules are located in the Column Colors category. Figure 5.12 shows the categories for color rules as they are provided by Ascolog Insight. In order to configure the properties of a color rule, select the appropriate color rule in the Functions tree and right-click. This will open the context menu where you must select Edit... to open the Color Rule dialog shown in Figure 5.13. 58/147 Figure 5.13: Configuring a color rule Press the Bkg... button to change the background color or the Text button to change the text color of the color rule. Both commands will open another dialog where you can select a color from a wide range of colors. If you press the Random button Ascolog Insight will randomly create a color scheme for the color rule. The selected color scheme will be displayed in the Color field. The Apply rule to field specifies to which column a color rule is applied. If there is a filter expression the color rule might only be applied to some cells of the column. When there are several color rules for the same cell the Z-Order field specifies the visibility of a color rule in relation to the other color rules. If several color rules have to be applied the one with the highest priority is used. The following precedence settings are possible: Lowest, Lower, Medium, Higher, Highest, Automatic selection The values of the Apply rule to and Z-Order fields are set by Ascolog Insight when a color rule is created. You usually don't have to change them. However, if you need a different color setting than what is proposed by Ascolog Insight you can manually change the Z-Order to a different priority to set a color to the foreground or background. As already mentioned above there are three standard categories of color rules in Ascolog Insight. What values of the Apply rule to and Z-Order fields are used by default for these categories is shown in Table 5.8. Type Description Apply rule to Z-Order Color rule Color rule for specific cells of a Automatic selection Automatic selection Column color Color rule of a complete Column that was Lower column. Created by clicking on selected during column. the header of a column and by creation of the rule. selecting Color Rule. Merge color Color rule of a log file. Created by Ascolog Insight when a log Full row file is loaded. Table 5.8: Color rule types 59/147 Lowest Please not that in the Apply rule to field you can select the values Full row and FULLROW which are different. The value Full row refers to all columns, i.e. to all cells of a row (remember that a log record is represented by a row) whereas FULLROW is a single column that contains the complete log record. In the Apply rule to field Automatic selection means that Ascolog Insight will examine the color rule's filter expression to determine to which column the color rule has to be applied. If the color rule's filter expression consists only of one expression like a simple comparison (e.g. TYPE==Error) then it will apply the color rule to this column (TYPE in this example), otherwise the color rule will be applied to the first column which is mentioned in the filter expression, e.g. when the filter expression is (LEVEL==High) || (TYPE==Error) then the color rule will be applied to the column LEVEL. If two color rules have the same Z-Order the color of the color rule that was created later is on the foreground and visible. 5.7 Record Filter Record filters are analyze function which are used to display only those log records in the Log window that meet the criteria defined by a filter expression (e.g. only log records of type ERROR which contain the string “a problem occurred” in the description). Filter expressions are something different than record filters. Filter expressions are also used by other analyze functions. Since a record filter is an analyze function the usual ways for creating analyze functions can be applied. How to create analyze functions is described in 5.3 How to create Analyze Functions (!) and 5.4 Filter Expressions (!). By default record filters usually will be created in the category Record Filter . An exception are quick filters. A fast way to create a record filter is provided by the menu command Quick filter... which opens a dialog where you can create the filter expression for a quick filter. It is the same dialog that is described in 5.4 Filter Expressions (!). These filters will be created in the Quick Filter category. Both categories are shown in Figure 5.14. 60/147 In the Edit menu or in the Toolbar you can find the commands Enable Filter and Disable Filter. These commands can be compared with a master switch which turns off electric power everywhere even though other switches are still turned on. Moreover, a master switch does not change the state of other switches. When you select Disable Filter all record filters even those that have ticked check boxes in the Functions tree are disabled. However, their current state, which is indicated by the check boxes next to their nodes in the Functions tree, is kept. This means that when you select Enable Filter all record filters that were active before pressing Disable Filter are active again. Whether Enable Filter or Disable Filter is currently active can best be seen in the Toolbar. If Enable Filter is active the Enable Filter button is gray and cannot be pressed. If Disable Filter is active the Disable Filter button is gray and cannot be pressed. If you think that your record filters don't work check if you have activated by accident Disable Filter. 5.8 Tabs Tabs are different views on the log records to be analyzed. The displayed log records are selected on the basis of the tab's filter expression. Ascolog Insight creates a new tab for each log file that is loaded. All the tabs can be accessed via the Tab list. You can change the order of the tabs in the Tab list by dragging a tab to a different position. The Main tab is always available and the Main tab's filter expression is set to display a merged view of all loaded log records. When you cannot see a log record in the Main tab check if any record filters prevent these log records from being displayed. Since a tab is an analyze function the usual ways for creating analyze functions can be applied. How to create analyze functions is described in 5.3 How to create Analyze Functions (!) and 5.4 Filter Expressions (!). You can also add a new tab when you right-click on the Tab list and you select New... The Tab Settings dialog will be displayed where you have to configure the settings of the new tab. 61/147 Tab type Description Primary This is the type of the Main tab. Linked A linked tab links to a log file and displays the content of this log file. A linked tab has the following two properties: 1. A linked tab links to a color rule in the Merge Colors category. 2. A linked tab is removed when the linked log file is unloaded. If you want to create a linked tab for a file, right-click on the file to open the context menu and select Tab and Linked. A custom tab whose filter expression refers to a log file does not have the two properties explained above. Custom Persistent A custom tab is based on a filter expression customized by the user. This is a tab where the Persistent Tab check box in the Tab Settings dialog is ticked. Table 5.9: Tab types When you create a new tab the tab default settings will be used. In order to modify them open the Tab Default Settings menu in the View menu. In the Tab Default Settings menu you can either directly change most of these settings or you can select the command Options... to open the Tab Default Settings dialog. As the tab default settings are the same as the settings for a specific tab please read 5.8.2 Editing a Tab to learn more about default tab settings. There are different types of tabs. They are described in Table 5.9. 5.8.1 Switching between Log Records in Different Tabs Assume you are studying the log records of a tab which only displays log records of type ERROR. In the tab a log record (let's call this log record ABC) should be investigated further, i.e. the log records before and after the log record ABC should be analyzed. Therefore it's the best to switch to the position of the log record ABC in the Main tab. You don't have to manually find the log record ABC in the Main tab but you can use Ascolog Insight's Select commands. The commands can be found in the context menu of the Log window. They offer you an easy method for switching between the positions of a log record that occurs in several tabs. In order to use these commands select a log record by hovering over one of its cells and right-click to open the context menu. In the context menu choose the menu Select. The Select menu offers you the commands listed in Table 5.10 to switch between tabs. Command Available Description Select Main Tab Not available when the The location of the selected log record in the Main Tab is already Main tab will be displayed. Not available when the The location of the selected log record in its linked tab is already linked tab will be displayed. A linked tab links selected. Select Linked Tab selected. to a log file and displays the content of this log file. 62/147 Select Previous Tab Always available This command selects and shows the tab that was previously displayed. When you select this command the currently selected tab becomes the previously selected tab. Example: Tabs X, Y were displayed. Z is currently displayed. When you select this command the tab Y is displayed and the tab Z becomes the previously displayed command and not X, i.e. when you select this command again the tab Z is displayed again, i.e. using this command you can switch between two tabs back and forth. 63/147 Select Context Tab Always available The location of the selected log record in its context tab will be displayed. If the context tab does not already exist it will be created. When you hover over a cell and you create a tab the cell's data is used in the filter expression of the tab. Such a tab is called the context tab of the cell. File Always available In the Files window the focus is set to the log file from where the selected log record originates. The file with the focus will get an orange background color. Table 5.10: Tab switching commands 5.8.2 Editing a Tab In order to edit the settings of a tab right-click on the tab to open the context menu. Select the Edit... command. The Tab Settings dialog shown in Figure 5.15 will open. Figure 5.15: Configuring a tab 64/147 Table 5.11 describes the settings which can be configured in the Tab Settings dialog. Field Description Label Defines the name of the tab. Filter Defines the filter expression of the tab. Options Persistent tab To save a tab there are two steps necessary: This check box must be ticked and then the layout must be saved. Tabs that are not marked as persistent are not saved when the layout is saved. Use tab filter for (important): Let's assume there is a tab with the name XYZ. In order to use the filter expression of the tab XYZ for creating an analyze function or another filter expression the following steps are required: 1. Tick the check box of the element you want to create in the Tab Settings dialog of the tab XYZ (e.g. New record filter node for a new record filter, etc.). 2. Close the Tab Settings dialog and set the focus to the tab XYZ in the Log window. 3. You have to handle analyze functions and filter expressions slightly differently: • Analyze functions: Select a cell in the tab XYZ whose data should become part of the new analyze function's filter expression and right-click to open the context menu where you can select the new analyze function. • Filter expressions: Drag a cell's data, which should become part of the new filter expression, to the Functions tree and drop it at the desired position. 4. The newly created filter expression combines with a logical AND the data of the cell with the filter expression of the tab XYZ. New record filter node When ticked a new record filter is generated as described in Use tab filter for. A logical AND is used to combine the tab's filter expression with the filter expression of the new record filter. New bookmark node When ticked a new bookmark is generated as described in Use tab filter for. A logical AND is used to combine the tab's filter expression with the filter expression of the new bookmark. New Tab When ticked and a new tab node is generated as described 65/147 in Use tab filter for. A logical AND is used to combine the tab's filter expression with the filter expression of the new tab. New color rule node When ticked a new color rule filter node is generated as described in Use tab filter for. A logical AND is used to combine the tab's filter expression with the filter expression of the new color rule. New graph node When ticked a new graph is generated as described in Use tab filter for. A logical AND is used to combine the tab's filter expression with the filter expression of the new graph. New Expression When ticked a new filter expression is generated as described in Use tab filter for. A logical AND is used to combine the tab's filter expression with the new filter expression. Use tab color for new graph When ticked a new graph is generated as described in Use node tab filter for. The tab's color (this is the color used in the Tab list) will be used for the new graph in the Graph window. Otherwise the color of the selected cell will be used for the graph. Custom tab color Defines the tab color (this is the color used in the tab list). Use dedicated category A category (represented as folder in the Functions tree) will be dedicated to the tab, i.e. the tab will only be influenced by analyze functions from the dedicated category, the analyze functions of the dedicated category will not have any effects on other tabs and any other analyze functions outside of the dedicated category won't have any effect on the tab. You can specify the name of the dedicated category in the text box. In order to create the dedicated category nodes in the Functions tree you have to execute the command Update Dedicated or Update All which are located in the View/Tab Categories menu. Use dedicated shared category A dedicated shared category works in the same way as a dedicated category but it is shared among several tabs. You can specify a name for a dedicated shared category in the text box. All tabs which specify the same dedicated shared category name are dedicated to the same category In order to create the dedicated category nodes in the Functions tree you have to execute the command Update Dedicated Shared or Update All which are located in the View/Tab Categories menu. 66/147 Use dedicated graph When ticked and a new graph is generated as described in Use tab filter for. The specified graph window will be used for the new graph. Otherwise the graph window which is specified in the Graph Default Settings menu will be used. Table 5.11: Configuration options of a tab A dedicated and dedicated shared category in the Functions tree are shown in Figure 5.16. It is not possible to edit the Main tab. Linked tabs can be edited but it is recommended not to change them because they will become custom tabs and the links will be lost. The message box shown in Figure 5.17 will be displayed to warn you about this. Figure 5.17: A message box that warns that a linked tab will lose its links If you need to modify a tab which provides a view on a file it is recommended to create an additional custom tab and to work with this tab. 5.8.3 Removing Tabs In order to remove a tab select the tab by right-clicking. This will open the context menu where you have to select Remove. Any files that are associated with the tab will not be unloaded. If you want to remove all tabs of a type in one step, right-click on the Tab list but not directly on a tab and select Remove/Linked, Remove/Custom, Remove/Persistent. This will remove all tab s of the type linked, custom or persistent. In order to see the types of the used tabs, right-click on the Tab list to open the context menu and select the List... command. This will open the Tab list dialog shown in Figure 5.18. You can also remove all tabs except for the Main tab by selecting Remove All. 67/147 5.8.4 Modifying the Tab list In order to edit the Tab list right-click on the Tab list and select List... or select Tab List... in the View menu. This will open the dialog shown in Figure 5.18: Select the Add button to add a new tab at the end of the list or the Insert button to insert a tab at the currently selected position. The position in of the tabs in the Tab list dialog determines the position of the tabs in the Tab list. Both buttons will open the Tab Settings dialog, which is described in 5.8.2 Editing a Tab, so that you can configure the new tab. Another way to create a new tab is the AND button. Select two or more tabs in the list and press the AND button. This will create a new tab whose filter expression is a combination of the filter expressions of the selected tabs using a logical AND. If you press the OR button the filter expressions are combined using a Figure 5.18: An overview of all tabs logical OR. When you press the Copy button the currently selected tab is cloned and added to the list. If you want to modify a tab, select the tab and press the Edit button which opens the Tab Settings dialog. You can use the Clear button to delete all tabs except for the Main tab. When you press the button Remove the currently selected tab will be deleted. Use the Arrow keys to move 68/147 the currently selected tab to a different position in the list This will also change the position of the tab in the Tab list which is below the Log window. Press the Restore button to undo all changes made to the order of the tabs. If you want to save your changes press the OK button. Press Cancel to discard all changes (sort order, new tabs, changed tab settings). 5.9 Bookmarks Bookmarks are intended to mark log records. The purpose is to help you to quickly find log records again. Log records that are bookmarked are always displayed in the Log window even if there are record filters that filter out these log records. Such log records will get a light gray font color and a white background color. Since a bookmark is an analyze function the usual ways for creating analyze functions can be applied. How to create analyze functions is described in 5.3 How to create Analyze Functions (!) and 5.4 Filter Expressions (!). When you have a look at the Edit menu you will notice that there are two ways of creating a bookmark. The method Edit/New Node/Bookmark will create a bookmark node in the Functions tree but without a filter expression, i.e. the user still has to create the filter expression. You can create filter expressions (like TYPE==Error) to create bookmarks for a group of log records. A common use case is to create a bookmark for a specific log records. Therefore you select the log record in the Log window that should be bookmarked. Then you open the menu and select Edit/Bookmark/Toggle (keystroke combination Ctrl+F2). This will create a bookmark with an expression filter where the expression filter identifies the selected bookmark by using the log record's ID (column UID) and the log record's source file (column FILE). Disabling such a bookmark works the same way as enabling: Select the log record marked with the bookmark and select Edit/Bookmark/Toggle or use the keystroke combination Ctrl+F2. As a consequence, the bookmark icon will disappear but the bookmark node will not be deleted. In order to delete a bookmark you have to delete the bookmark's node in the Functions tree. By default the bookmarks created with Toggle are created in the Record Bookmarks category, the other bookmarks are created in the Bookmarks category. This is shown in Figure 5.19. Ascolog Insight also offers the possibility to navigate between bookmarks. Next to the Toggle command in the menu there is the Go To menu which contains the following commands: Go To Next Bookmark, Go To Next Bookmark Region, Go To Previous Bookmark and Go To Previous Bookmark Region. You can also find these commands in the Toolbar. A bookmark region is 69/147 formed by log records that are consecutive and which have a bookmark with the same filter expression. The first log record of a region that is found in the search direction gets the focus. 5.10 Graphs The analyze function graph supports the visualization of logged information. Since graphs belong to the analyze functions of Ascolog Insight and thus have nodes in the Functions tree. The application provides four graph windows. A graph window can display several different graphs. Since a graph is an analyze function the usual ways for creating analyze functions can be applied. How to create analyze functions is described in 5.3 How to create Analyze Functions (!) and 5.4 Filter Expressions (!). By setting a filter expression it is possible to configure what should be displayed by a graph. Fixed samples are also graphs and this is indicated by the graph icon (see Figure 5.20): In the Graph window the currently selected log record of the Log window is marked with a thick vertical red line. The “cursor” is a thin dashed vertical red line. You can only select log records in the Graph window which are not filtered out in the Log window. Figure 5.21: A black frame marks those log records in the graph which are currently displayed in the Log window In the options you can configure that the records which are displayed in the Log window will be marked in the graph with a black frame as shown in the Figure 5.21. The name of the option is ShowDisplayedRecordFrame and is located in the Display category. The x-axis of a graph in Ascolog Insight always displays timestamps. These timestamps must follow the ISO standard, i.e. they must have the format: 70/147 YYYY-MM-DD hh:mm:ss,mmm (Y=Year, M=Month, D=Day, h=hour, m=minute, s=second, m=millisecond) If your timestamps have a different format you can use a custom columns definition to convert your timestamps to ISO timestamps. The converted timestamps will be available in their own column: ISO_TIMESTAMP. For more information please refer to 10 Custom Columns Definitions (CCD). There are two types of graphs: timeline and column 5.10.1 Timeline Graphs A timeline graph (see Figure 5.22) visualizes the occurrence of certain log records. In the figure you can see three timelines: 0, 1, and 2 but you can draw many more. Each triangle and square represents a log record. The lines between the sample shapes are just for better visibility but do not represent any data. At the bottom of the Graph window you can see a legend that displays the names of the analyze functions which are used in the Functions tree. Figure 5.22: Example of a Timeline graph Visualizing the Occurrence of Log Records In order to visualize all log records of a log file either select the file in the File window or a cell in the FILE column in the Log Window and right-click to open the context menu. In both context menus you select Graph/Timeline/Auto or Graph/Timeline/<timeline number> (e.g. Graph/Timeline/2). If you want to visualize all log records of all files you have to use the wildcard for the filter expression of the graph. 5.10.2 Column Graphs A column graph (see Figure 5.23) visualizes data which is retrieved from the log records, e.g. the number of concurrent users, etc. The data for column graphs must be provided in their own columns. In Figure 5.23 the y-axis displays the memory usage in percentage terms. 71/147 Figure 5.23: Example of a Column graph displaying the memory usage in percentage terms 5.10.3 Graph Window Properties You can configure in the Graph Display Options... in the View menu several properties of the four Graph windows (see Figure 5.24). Figure 5.24: Graph Display Options dialog When you want to configure the appearance of the Graph windows as in Figure 5.22 or Figure 5.23 you have to tick all check boxes except Draw vertical. 72/147 Figure 5.25: A vertical Graph window is aligned besides the Log window When you want to visualize the occurrence of log records in a graph it is quite convenient to arrange the graphical representation of the records vertically next to the records of the Log window as shown in Figure 5.25. When you want to do this tick the Draw vertical check box. 73/147 5.10.4 Graph Properties Figure 5.26: Graph dialog In order to configure the visual properties of a graph, select the appropriate graph node in the Functions tree, right-click to open the context menu and select the Edit... command. The Graph dialog which is shown in Figure 5.26 will be opened. The properties that can be configured are described in Table 5.12. Property Description Window Specifies the selected Graph window that is Color Sets the color of the graph. You can select a used to display the graph. color from a color menu (Select...) or you can create a random color by pressing the Random button. Press the SHIFT key while selecting the Graph command in the context menu to generate a random color for a graph. Draw mode Defines what graphical elements are drawn in the Graph window. The following basic elements, which can be combined, are available: Lines: the graph function's adjacent output values are connected with lines Samples: graphical element like squares or 74/147 rectangles are drawn for each output value of the graph function. Moreover, when lines are drawn you can make them Alternating. See 10.2.3 Visualizing Call Sequences of Functions -Enter/Exit Logging for a use case of this feature. Line style Sets the line style, e.g. solid, dashed, etc. Sample shape Sets the style of the graphical elements of type Scale Scales the y-axis by multiplying the standard sample (see Draw mode property above): e.g. triangle, square, etc. values with the selected scale value, e.g. 500 * 10 when the scale value 10 is selected. Type Sets the type of a graph: timeline or column Timeline index Selects a timeline. Only available if the graph Column Selects a column whose data should be type timeline was selected. displayed. Only available if the graph type column was selected. Fixed samples Shows the number of fixed samples that are already set. The button Select… opens a dialog to add new fixed samples. No own node is created in the fixed samples category of the Functions tree and thus fixed samples created from the graph dialog cannot be accessed from the Functions tree. Use the Graph dialog and the Select… button to access previously created fixed samples. More about fixed samples can be found in 5.10.8 Fixed Samples. Z-Order Defines the visibility of the graph. If graphs overlap the one with the highest value in the ZOrder field is displayed on top. Table 5.12: Graph dialog properties In the View menu you can find the Graph Default Settings sub-menu. In this menu you can configure the default settings for a graph either in the Options dialog or directly in the menu. These settings are applied when a new graph is created. When the default settings are changed the new settings won't be applied to existing graphs. The graph default settings are a subset of the settings listed in Table 5.12. They cover setting the graph window, the draw mode, the line style and the sample shape. 75/147 5.10.5 Zooming Figure 5.27: Graph Zoom dialog It is possible to zoom in to an area of the graph by selecting a rectangular zoom area with the mouse. Left click on the Graph window to define the first corner of the rectangular zoom area and while pressing the left button drag the mouse pointer to the opposite corner of the zoom area. In order to zoom out right click on the Graph window and the last zoom will be undone. If you have zoomed in several times and you want to zoom out completely you either have to repeat zooming out several times by right clicking on the Graph window or you can press the Reset Zoom button in the Toolbar or select the Graph Reset Zoom command in the View menu. The Reset Zoom button is gray when no zoom is active. Another way to use zooming is to go to the View menu and to select Graph Zoom...This will open the Graph Zoom dialog (see Figure 5.27) where you can specify a zoom area based on timestamps (x-axis) and function values (y-axis). You also have to specify the Graph window where the zoom has to be applied to. 5.10.6 Adjusting Graphs This feature is helpful when comparing graphs from different Graph windows. In the View menu you can select what graphs should be adjusted by selecting their Graph windows (command Graph Adjust or Graph Adjust All). The scale of the x-axis in the selected Graph windows is adjusted so that the x-axis have the same timestamp as starting point and the same timestamp as ending point. So if you select Graph Adjust for Graph window 1 and Graph window 2 Ascolog Insight will use the following starting and ending points for the adjusted x-axis: starting point = min(Graph window 1 starting point, Graph window 2 starting point) ending point = max(Graph window 1 ending point, Graph window 2 ending point) 76/147 You can compare graphs more easily when you position their Graph windows exactly below each other. Then the same timestamps will also be positioned below each other. 5.10.7 Saving a Graph to a File Ascolog Insight offers the possibility to save a graph. Open the View menu and select Graph Save... This will open the dialog shown in Figure 5.28. In the Graph window drop down list select the graph window that should be saved. In the File name edit control you can specify the file where the graph will be saved. The supported file type is png. The extension has to be specified, too. Alternatively, you can press the “...” button to open a file browser dialog. In the Width and Height edit controls you can specify the dimensions of the graph in pixels relative to the upper left corner. If you specify zero in these fields the dimensions of the selected graph window are used. Figure 5.28: Saving a graph 5.10.8 Fixed Samples Fixed samples are special graphs. They are used to mark important points in time to facilitate the orientation in a graph, e.g. you could use two fixed samples to mark the beginning and the end of the time period which should be investigated or one fixed sample to mark an important event. Record filters do not influence fixed samples. Fixed samples can be added in two ways: • to an existing graph • as own graph with its own node in the Functions tree, independently from any “normal” graphs If fixed samples are added to an existing graph and the existing graph is deactivated or deleted the fixed samples are also deactivated or deleted. To add a fixed sample to an existing graph, select the appropriate graph in the Functions tree, right-click to open its context menu and press the Edit... button. This will open the Graph dialog shown in Figure 5.29. More information about this dialog can be found in 5.10.4 Graph Properties, Table 5.12. 77/147 Figure 5.29: Graph dialog Press the Select button next to the Fixed samples field to open the Fixed Samples overview shown in Figure 5.31. When you press the Add button in the Fixed Samples overview the Add Fixed Sample dialog is opened (see Figure 5.30). Figure 5.30: Add Fixed Sample dialog In order to add a fixed sample as own graph, the command Add fixed sample... from the Edit menu must be used. This will directly open the Add Fixed Sample dialog which is shown in Figure 5.30. When you add a fixed sample as own graph a Fixed Samples category node (if it does not already exist) and a Fixed Sample node are added to the Functions tree. 78/147 If you add a fixed sample (both ways) you have to provide a timestamp and a timeline index. The graphical element representing the fixed sample is created at the specified timestamp at the specified timeline index in the Graph window. Figure 5.31: Fixed samples dialog In order to see a list of all fixed samples go to the Functions tree, right click on a “normal” graph or a fixed sample graph and select Edit... The Graph dialog shown in Figure 5.29 will be opened. The Fixed samples field displays the number of fixed samples that were already created. Use the Timeline index drop down list to specify the timeline that should be used for these fixed samples. Press the Select... button next to the Fixed samples field to open the Fixed samples dialog (see Figure 5.31). The Fixed samples dialog provides an overview of the existing fixed samples. In this dialog you can also add, edit or remove the selected fixed sample or remove all fixed samples (Clear button). When you press the Add or Edit button the Add Fixed Sample dialog is opened. The Value column displays the index of the timeline that is used for displaying the fixed samples. 79/147 5.10.9 Troubleshooting • You might not have specified a timestamp column in the options. Open the Tools menu and select Options... Go to the Columns/TimestampColumn option and select in the Value drop down list a column that provides timestamp values. If the TimestampColumn is set to <default> you have to make sure that the option Columns/MergeColumn is set to a column that provides timestamps. • The timestamps don't have the ISO timestamp format. • You might have told Ascolog Insight in the analyze function graph to display the graph in a Graph window which is not open. • You might have zoomed in too much so that an area is displayed where there are no graphical elements. • The graph is hidden behind another graph. 5.11 Data Extraction (!) The Details window offers the possibility to extract data from log records. Data extraction is based on custom columns definitions. They are described in detail in 10 Custom Columns Definitions (CCD). When you supply all the information needed for data extraction and you press the OK button in an Extract Text or Extract Data dialog Ascolog Insight will create the appropriate commands in the layout's custom columns definition which extract the specified data. This means that instead of using the data extraction dialogs you could also create the appropriate commands by hand. 5.11.1 Extract Text (!) This section explains how to extract a text. In the Details window the context of the text which should be extracted must be selected. The context consists of the extractable text and the preceding and succeeding texts which are required to unambiguously define the extractable text. All log records that contain the specified context are considered when extracting texts. Example 5.1 is used to illustrate what should belong to a context. 2013-10-10 10:12:25,000 I 0x1BE4 Database initialization time: 1234 ms. 2013-10-10 10:12:26,000 I 0x1BE4 Average transaction time: 3333 ms. 2013-10-10 10:12:27,000 I 0x1BE4 Application initialization time: 543 ms. Example 5.1: Extracting the initialization times 80/147 When you want to extract all initialization times as they are shown in Example 5.1 the text “initialization time: 1234 ms.” is an appropriate context because this ensures that both the database and application initialization times can be extracted and that the average transaction times are not included. In order to identify all initialization times it is important to exclude “Database” and “Application” from the context. In the Details window select e.g. “initialization time: 1234 ms.” and right-click to open the context menu of the Details window. In the context menu you have to select Extract and Text... to open the Extract Text dialog. Figure 5.32: Extract Text dialog after opening The Extract Text dialog after opening is shown in Figure 5.32. Some of the fields are still empty. The next step is select the text that should be extracted from the context in the Select the text part that should be extracted if it appears in the specified context field as shown in Figure 5.33. This will immediately update the Prefix, Extract and Postfix fields. If the options are set as desired the text extraction can be started by clicking Save & Reload. If only the Save button is pressed the layout will be adapted but the changes are not visible in the user interface. The text extraction can also be canceled. 81/147 Figure 5.33: Extract Text dialog after selecting the text part to extract The fields of the Extract Text dialog are explained in detail in Table 5.13. Property Description Select the text part that should be extracted if it Displays the selected context from the Details appears in the specified context window. The text to be extracted must be selected in this edit control. This will update the Prefix, Extract and Postfix edit controls. Column Specifies the column to be used for displaying the extracted text. You should provide a meaningful name here. Prefix Contains the text part of the context preceding the extracted text. Regular expressions might be used. Extract Contains the extracted text. The type of the extracted text is one of the following: Time, DateTime, Pathname, Filename, Alphabetic, Alphanumeric, Text, MultiText, String, Custom The types are defined in the layout's grammar definition file using regular expressions. Postfix Contains the text part of the context succeeding the extracted text. Regular expressions might be used. Format Defines the format of the extracted text in its column. Unformatted - the extracted text is left aligned, no padding Right-aligned, space padding - the extracted text is right aligned and padded with spaces on its left side Right-aligned, zero padding - the extracted text is right aligned and padded with zeros on its left side Left-aligned, space padding - the extracted text is left aligned and padded with spaces on its right side Left-aligned, zero padding - the extracted text is left aligned and padded with zeros on its right side The number of padding characters can be configured. Table 5.13: Extract Text options 82/147 5.11.2 Extract Column (!) The information in a log might be structured in columns as shown in Example 5.2, e.g. the pieces of information might be separated by commas or other delimiters (column separated values, CSV). These columns will be called CSV columns in this section and must not be confused with the columns used in Ascolog Insight to display log data in the Log window. These columns will be called log window columns in this section. The following steps are required to extract CSV columns from a log and to display them in their own log window columns. In the Details window all information that belongs to such CSV columns should be selected even if some information might not be of interest. This means that in Example 5.2 the complete line “Type;Time;Code1;Code2;Amount” or “1;2013-03-13 15:39:33,767;0x18AC;0x1928;34;” should be selected even if the CSV column which contains e.g. the timestamps might not be of interest. Type;Time;Code1;Code2;Amount 1;2013-03-13 15:39:33,767;0x18AC;0x1928;34; 1;2013-03-13 15:39:33,767;0x18AC;0x1928;34; 1;2013-03-13 15:39:38,304;0x1230;0x1864;33; 1;2013-03-13 15:39:38,350;0x1230;0x1864;33; Example 5.2: A log containing comma separated values After selecting a line right-click on the Details window to open the context menu. In the context menu you have to select Extract and Column to start the Extract Column dialog which is shown in Figure 5.34. In the Select column field the selected text from the Details window is displayed. Figure 5.34: Extract Column dialog after opening 83/147 The first step is always to select the separator which is used to separate the CSV columns in the in the Separator field. The next step is to select a CSV column that should be extracted from the Select column field. This will update the Selected Column, Prefix and Extract fields. If the fields are set as desired the data column extraction can be started by clicking Save or Save & Reload. If only the Save button is pressed the appropriate command will be created in the layout's custom columns definition but the changes won't be applied to the Log window and the Extract Column dialog will stay open. This is useful when several CSV columns have to be extracted. Press Save & Reload after you specified the last CSV column that should be extracted to reload the layout and apply the changes to the Log window. If you press the Close button the dialog is closed. The Group... button enables you to extract several CSV columns in one step. It opens the Column Group dialog shown in Figure 5.35 where you can define what CSV columns should be extracted and their names in the Log window. The Column Group dialog makes a suggestion for the names of the log window columns. This is most useful when you selected a line that contains descriptive headers of the CSV data. Select a column and press the Edit... button to change the column's name. Select the columns you want to keep and press the Keep button to keep these columns and to remove the non-selected columns. If you press the Remove button the selected columns will be removed and the non-selected columns will be kept. You can select multiple columns by pressing the CTRL key while selecting columns. It is also possible to define the format of a column as described in Table 5.14 by pressing the Format... button. In order to execute the Group command the Save & Apply button in the Column Group dialog must be pressed. Figure 5.35: The Column Group dialog allows extracting several CSV columns in one step. 84/147 The fields of the Extract Column dialog are described in detail in Table 5.14. Property Description Selected column After the dialog pops-up this control displays the data selected in the Details window. The text that is selected in the ”Select the text part that has to precede the first column” edit control will be removed from this control. The CSV column to be extracted must be selected in this edit control. This will update the Prefix and Extract edit controls. Only one CSV column can be selected in one step. If several CSV columns should be extracted in one step the Group... command must be used. Select the text part that has to precede the first After the dialog pops-up this control displays column (optional) the data selected in the Details window. The user can specify a text that must precede the first column. This will simplify the identification of the CSV columns in some cases and remove unnecessary text. The selected text will be removed from the Selected Column edit control. It will be added to the Preceding Text edit control after the data to be extracted is selected in the Selected column edit control. Example line: Running applications: App1, App2, App3 Select “Running applications” as preceding text. This will remove the text “Running applications” from the columns to be created and identify the CSV columns. Column Defines the column and its name in the Log window to be used for displaying the extracted data column. Separator Defines the delimiter that separates the CSV Prefix Specifies the data that precedes the data to be columns. extracted. Regular expressions might be used. Extract Specifies the data to be extracted. The data is Format Unformatted - the extracted text is left aligned, no padding Right-aligned, space padding - the extracted text is right aligned and padded with spaces on defined using regular expressions. 85/147 its left side Right-aligned, zero padding - the extracted text is right aligned and padded with zeros on its left side Left-aligned, space padding - the extracted text is left aligned and padded with spaces on its right side Left-aligned, zero padding - the extracted text is left aligned and padded with zeros on its right side The number of padding characters can be configured. Table 5.14: Extract Column options 86/147 6 Searching and Monitoring Log Records 6.1 Searching In order to start a search open the Edit menu and select Find... The dialog shown in Figure 6.1 will be displayed. Figure 6.1: Find dialog Enter the search term into the Find what edit control. You can configure Find options to improve the refine your search: • Match case • Match whole word • Search up • Regular expression You can combine these options if the combination makes sense. After the settings are correct press the Find Next button to start your search. Ascolog Insight will search in the view of the Log window (e.g. Main tab) which is currently active and the search will start at the position of the log record which is selected. The log record of the current successful match will be displayed in the Details window and the search term is highlighted in red. Moreover, in the Log window the current log record which contains the search term gets a white background. In order to search for the next log record that contains the search term press Find Next in the Find dialog, the key F3 or open the Edit menu and select Find Next. When the end of the Log window is reached the message box shown in Figure 6.2 is displayed and the search continues at the beginning (when searching forwards) or the end of the Log window (when searching backwards). 87/147 Figure 6.2: Search reached the end of the Log window 6.2 Monitoring - Find in Files Since only those log records are retrieved which match the filter criteria Find in Files is a good method to deal with large log files. Figure 6.3: File monitoring configuration When you want to monitor log records that match a certain filter from log files which are periodically updated (new log records are added while the file is open in Ascolog Insight) the 88/147 command Find in Files is the best way to do this. Find in Files opens a configuration dialog which provides a convenient way to configure monitoring log records. Go to the Edit menu and select the Find in Files command to open the Find in Files dialog which is shown in Figure 6.3. When you press the Find button in the Find in Files dialog Ascolog Insight registers all files that match the file filter and from these files it loads those log records that match the search criteria. Enter the search term in the edit control Find what. You can specify options to refine your monitoring in the Find options. You can combine these options if the combination makes sense. • Match case • Match whole world • Regular expression You can also specify a record filter so that only log records that match this filter are searched (e.g. log records that contain the string “abc”). Existing record filters from the Functions tree can be used by pressing the Apply Active Record Filter button. Strings must be encompassed by double quotes (i.e. “abc” and NOT abc) in a filter expression. In the Look in edit control you specify the folder where to look for files that should be included in the monitoring. However, you can also specify a file. Check the Include sub-folders box to include sub folders in the monitoring. The type of the files to be monitored is provided in the Look at these file types field. You can specify several file types by separating them with a semi-colon. If Tail is set to zero the complete file is monitored. If Tail is set to a positive value N only the last N mega-bytes of the files are considered. Find in Files works closely together with the refresh feature of Ascolog Insight so in order to process log records which were added to monitored files after the start of the monitoring open the File menu and select the Refresh command or you can also schedule a refresh task as described in 9 Task Scheduling. Using the Find in Files dialog again instead of Refresh won't work. When a file is already loaded in Ascolog Insight it will be ignored after pressing the Find button of the Find in Files dialog. The File Information dialog shown in Figure 6.4 will pop-up to inform you about this. You must confirm that the file is ignored by pressing the Ignore button. When you want to make this confirmation for all loaded files press the Ignore All button. When you press the Abort button Find in Files is aborted. If you want to change the monitoring configuration for loaded files (e.g. a different value in the Find What field) you must first unload these files. Figure 6.4: File Information dialog 89/147 All loaded (registered) files are listed in the Files window. You will find useful information about the monitored log records there. In the Retrieved Records column of the Files window you will see the number of log records that were loaded from a file. When a file was loaded due to Find in Files the value in the Retrieved Records column is equal to the number of log records that contain the search term (the number is zero when no log record matches the Find What criteria). The Processed Records column will display the total number of log records in a loaded file. The Files window will show all files that match the file criteria even if the search term was not found in some of the files. 90/147 7 Options Many features of Ascolog Insight can be adapted to your requirements. In order to change the configuration of features open the Tools menu and select Options... to open the Options dialog shown in Figure 7.1. Options can be valid only for a single layout or for all layouts (i.e. application-wide). Figure 7.1: Options dialog In order to change an option select the option in the tree on the left side of the Options dialog and select it. The options are organized by their purpose in categories (e.g. Display, File, Columns, etc.). The details of the selected option are shown on the right side of the Options dialog. In the details you can configure the option by changing its value in the Value drop down list. For more information about an option please have a look at the Description field. For some options you can find additional information in this documentation in the chapter related to the option. 91/147 8 Refreshing the Log Window 8.1 How to Keep the Displayed Logs Up-to-date Logs might change while they are loaded in Ascolog Insight. Ascolog Insight is capable of keeping the displayed information from logs up-to-date. There are the following modes to refresh the Log window: 1. Automatic refresh: the application refreshes after a configurable time period elapses 2. Semi-automatic refresh: the application displays out-of-date icons in the Files window but does not refresh the Log window. In order to refresh the Log window the user must trigger a refresh. 3. Manual refresh: the application does not check if a refresh is necessary, the user must trigger the refresh. The refresh mode of Ascolog Insight is configured in the Options dialog (option File/RefreshMode). The refresh mode can also be changed from the File menu (commands Enable Auto Refresh and Disable Auto Refresh). The File menu commands are intended to enable the user to quickly switch between the full-auto and semi-auto respectively the full-auto and manual refresh mode. Table 8.1 subsumes how the application refreshes logs dependent on the refresh mode set in the Options dialog (File/RefreshMode) and the selected command in the File menu: Configured Refresh Selected File menu Status (enabled or mode in the Options, command (currently disabled) of command File/RefreshMode selected command Applied Refresh mode Refresh in File menu appears in gray) FULLAUTO Enable Auto Refresh enabled (always!) Automatic FULLAUTO Disable Auto Refresh enabled when logs are Semi-automatic out-of-date, otherwise disabled SEMIAUTO Enable Auto Refresh enabled (always!) Automatic SEMIAUTO Disable Auto Refresh enabled when logs are Semi-automatic out-of-date, otherwise disabled MANUAL Enable Auto Refresh enabled Automatic MANUAL Disable Auto Refresh enabled Manual Table 8.1: Refresh mode options You have the following possibilities to manually refresh the Log window: Refresh and Force Refresh. Both commands are located in the File menu. Refresh only updates a log if the modification date of the log file is newer than the date of the last refreshing. As a consequence, a 92/147 log is not unnecessarily loaded again when it did not change. However, you can override this check by triggering the command Force Refresh. Manual refreshing is useful when logs are accessed over slow network connections and even the check for a newer modification date is slow. 8.2 Refresh Options To configure the refresh options go to the Tools menu and select Options. The refresh options are located in the category File. The available options are described in Table 8.2. Refresh options Description File/RefreshMode Sets the log refresh mode: FULLAUTO: Automatically updates the Log window. SEMIAUTO: Displays an out-of-date icon in the Files window but does not update the Log window. MANUAL: You have to trigger a refresh manually. Neither updates the Log window nor displays an out-of-date icon in the Files window when a log becomes outdated. File/RefreshSelection Defines what record is selected after refreshing the content of the Log window. KEEP_SELECTION: The currently selected log file keeps the selection. TOP_SELECTION: The record on the top is selected. BOTTOM_SELECTION: The record at the bottom is selected. This is useful when the record shall be selected that was most recently added. This option can also be configured in the Views menu, sub-menu Refresh Record Selection: Keep = KEEP_SELECTION Top = TOP_SELECTION Bottom =BOTTOM_SELECTION File/RefreshTimerInterval Sets the time interval, in milliseconds, between two refreshes for the Automatic refresh mode and for the Semi-automatic refresh mode. File/RefreshEnforceReload FALSE: Ascolog Insight refreshes the content of the Log window when the content's last refresh was before the last modification of one of the log files. TRUE: Ascolog Insight refreshes the content without checking any modification times. File/RefreshEnforceRecalc Enforces recalculation upon refreshing (see 93/147 11.3.1 Merge Records and Recalc Records). Table 8.2: Refresh options 8.3 How to Test Refreshing Read this when you manually add log records to a file to test refreshing. When files are opened they are mapped into memory. This means that the data in the memory and the data in the file influence each other because they are synchronized by the operating system. Logs should only be added to the end of a file. If they are added before the end, the mapping is destroyed and Ascolog Insight displays scrambled logs. 94/147 9 Task Scheduling 9.1 An Introduction to Task Scheduling You can configure Ascolog Insight to execute the following tasks automatically: 1. Search – finds log records in files as described in 6.2 Monitoring - Find in Files 2. Refresh – updates the information that is displayed in the Log window (see 8 Refreshing the Log Window) 3. Convert – converts and exports log records to a different format or to a different location (file, database, etc.) as described in 14 Exporting Logs The command Refresh from Task Scheduling and the commands Enable Auto Refresh/Disable Auto Refresh in the File menu do not use the same timer. You can schedule a search and refresh task only once since it makes no sense to schedule them more than once. However, you can schedule several convert tasks because you might want to convert to different log formats or export log records to different places (files, database server, syslog server, etc.). In the Tools menu you will find the task scheduling commands described in Table 9.1. Menu Command Purpose Scheduled Task... Opens the Scheduled Tasks dialog where you can configure the tasks that you want to schedule. Enable Scheduled Tasks Starts the execution of the scheduled tasks Disable Scheduled Tasks Stops the execution of the scheduled tasks Table 9.1: Task scheduling commands In the Options dialog, in the category Misc you can specify in the AutoStartScheduledTasks field if the scheduled tasks shall be automatically executed when Ascolog Insight is started. 9.2 Scheduled Tasks Dialog In the Scheduled Tasks dialog you can find the Task list. The search and refresh tasks are always displayed in the Task list. To add a convert task click on the Add or Insert button and provide all necessary information like you would do it in the Conversion dialog (see 14 Exporting Logs). The Reset button is only available when the search or the refresh task is selected. It resets all configuration settings of the currently selected task and sets the trigger to inactive. The Clear button resets the search and refresh tasks and removes all convert tasks from the Task list. The Remove button is only available when a convert task is selected. It removes the selected convert task from the Task list. 95/147 For each task you must specify a trigger, i.e. the condition when a task is executed. The trigger status is displayed in the Task list in an own column next to the task. The available triggers are listed in Table 9.2. Trigger Description Inactive No trigger is selected for the task. Suspended Suspends the triggers that are set for a task. Periodically Periodically executes a task using the provided schedule. Upon initialize Triggers a task after loading a layout or when the Scheduled Tasks dialog is finished by pressing the OK button. Upon search Triggers a task when a search is completed. Upon data changed Triggers a task when the data that is displayed in the Log window changes Table 9.2: Triggering options for scheduling tasks You can also set several triggers for one task (e.g. Upon initialize and Periodically). Note that some triggers are not available for all tasks (e.g. the Upon search trigger is not available for the search task). All you have to configure for the refresh task is the trigger. To configure the other tasks select the appropriate task and press the Edit button. How to configure a convert task is explained in 14 Exporting Logs. For a convert task the Config column of the Task list displays what is selected in the Encoding field in the Conversion dialog. The Path column denotes the destination of the conversion. If for the conversion a filter is defined it is displayed in the Filter column. How to configure a search task is explained in 6.2 Monitoring - Find in Files. For a search task the Config column displays the file extension that are considered in the search. The Path lists the folders that are included in the search. The Filter column displays what is specified as the record filter for the search. 96/147 10 Custom Columns Definitions (CCD) 10.1 Working with Custom Columns Definitions This chapter covers advanced topics related to custom columns definitions. Most users will be able to work with Ascolog Insight effectively without reading this chapter. However, before you study this chapter please read chapter 4 Working with Logs and Layouts which is important forch understanding this chapter. Besides columns created by the nonterminal symbols of a grammar definition Ascolog Insight also knows custom columns which are generated by a script which is called custom columns definition. The commands which can be used in this script are documented in the application's Help Topics, chapter CCD Script Commands. Custom columns are added by the user in order to extract additional meaningful information from the logs. A custom columns definition of a layout can consist of several files and among these files there is one file that is the “main” file. The execution of a script always starts in the “main” file (analogous to the Main function in many programming languages). Open the Layout menu and select Edit... to open the Edit Layout dialog shown in Figure 10.1. Figure 10.1: Editing a layout The field Custom column definition file name of the Edit Layout dialog displays what file is the “main” file. The “main” file can be changed with the Save & Apply command from the File menu of the Custom Columns Definition dialog (see Table 10.1, File menu description below). A custom columns definition is saved in a custom columns definition file which has the file extension “.cdf”. In order to create a custom columns definition open the Tools menu and select Customs Columns Definition... which opens the editor shown in Figure 10.2. 97/147 Figure 10.2: Custom Column Definition editor In the editor you can use Ascolog Insight's commands to create custom columns definitions. Commands are displayed in blue in the editor (e.g. map and xtr). Open the Help menu and select Info to get a list of the available commands with a description. Macros can be used to combine commands in order to solve more complex problems (in Figure 10.2 TIME_SHIFT is such a macro). When in a custom columns definition a macro is found the macro is replaced by its definition (like in the programming language C). In the Edit menu you will find some useful functionality for working with a custom columns definition file. Select the Find... command to specify a search in the currently loaded custom columns definition. Use Find Next to navigate to the next occurrence of the search term. If you change the custom columns definition you can use Undo/Redo to undo/redo any changes in the script. File menu Description Load Loads a custom columns definition file (file extension “.cdf”). A previously loaded custom columns definition file will be closed and any changes that were not stored will be discarded without any warning. Save & Apply Saves the current script to the custom columns definition file displayed in the title bar, makes this file the “main” custom columns definition file and subsequently applies the script. Save As Saves the current script to a custom columns definition file. The Save CCD File dialog will be opened where you have to provide a name for the custom columns definition file. Recent Files Displays the previously loaded custom columns definition files. Select a file to open it. Exit Closes the Customs Column Definition dialog without saving any changes. Table 10.1: Commands of the File menu 98/147 In the File menu you will find the commands listed in Table 10.1 to work with custom columns definition files. It is important to know how custom columns definitions are organized respectively stored in the file system. This is is shown in Figure 10.3. Where you find the data folder is described in 2.3.1 Installing with XCOPY (!) respectively 2.3.2 Installing with the Setup (!). Folder names are in bold, files are displayed in a normal font in Figure 10.3. Folders and files <Application's data folder> e.g. C:\ProgramData\AscologInsight or C:\MySpecialPathXYZ LAYOUTS TestLayout TestLayout.cdf _SHARED_ Columns basecolumns.cdf extendedcolumns.cdf Macros basemacros.cdf sdkmacros.cdf Figure 10.3: Custom columns definition files and folders If a custom columns definition should be valid for all layouts it must be added to the file “basecolumns.cdf” or “extendedcolumns.cdf”. If a custom columns definition should be valid only for a specific layout it must be added to the layout's custom columns definition file. The name of this file normally consists of the layout name and the file extension “.cdf”, i.e. “TestLayout.cdf” in the example above (see Figure 10.3). Custom columns definition files can be split and stored in several other files which are not listed in the example. All files that should be used by a layout must be included using the #include directive, e.g. the file “basecolumns.cdf” must be included in the layout's custom columns definition file when it shall be used by the layout. In order to include a file open the Edit menu and select Includes... to open the Includes dialog shown in Figure 10.4. 99/147 Figure 10.4: Including directories containing custom columns definition files Add the path where the file is located that has to be included to the Path name edit control. It is recommended to use relative paths especially when you used the XCOPY installation method. This ensures that you can copy Ascolog Insight to a different folder without having to modify the include paths again. If you use a relative path it is relative to the layout folder of the currently loaded layout (e.g. folder “TestLayout” in the file structure given above). It is also possible to specify absolute paths. Instead of directly entering a path you can also use the button with the three dots to start the Browse For Folder dialog. In the Browse For Folder dialog you can navigate to the path you want to add. However, in this case absolute paths will be used. If the path in the Path name edit control is correct press the Add button to add the path to the Additional include directories list. Select a path in this list and press the Remove button to delete it from the list. If all settings are correct press the OK button to quit the dialog and save the settings. Some functionality (e.g. certain custom columns definitions, macros) has to be enabled before it can be used because of performance reasons. Creating certain custom columns consumes a lot of processing power and memory of your computer. In order to enable/disable functionality go to the Edit menu and select Defines... to open the Defines dialog which is shown in Figure 10.5. Insert the name of the define (e.g. USE_FULLROW_COLUMN) that controls the functionality you want to 100/147 enable/disable in the Name field and set the appropriate value in the Value field (very often it's the value 1). In order to figure out what define has to be used you might have to browse the custom columns definition files (e.g. “basecolumns.cdf” or “extendedcolumns.cdf”). When you press the Set button next to the Defines or Undefines list the define will be added to the appropriate list. In order to delete a define/undefine select it and press the Remove button. Figure 10.5: Defines When all settings are correct press the OK button to save the settings and exit the dialog. 10.2 Predefined Custom Columns 10.2.1 Custom Columns Overview The files “basecolumn.cdf“ and “extendedcolumns.cdf” provide predefined custom columns which are very useful for analyzing log files. The columns must be activated by setting the appropriate defines in the Defines dialog. The available predefined columns are described in Table 10.2. 101/147 Columns of file Description FULLROW Defines: USE_FULLROW_COLUMN=1 basecolumn.cdf Source file: “basecolumn.cdf” Description: The FULLROW column contains a full log record by combining the content of the HEADER column and the TEXT column. UID Defines: USE_UID_COLUMN=1 Source file: “basecolumn.cdf” The UID column contains a unique numerical identifier for each log record. FMT_TEXT Defines: USE_FMT_TEXT_COLUMN=1 and FMT_TEXT_COLUMN=1, Source file: “extendedcolumns.cdf” The FMT_TEXT column uses as input the TEXT column. It's purpose is to improve the readability for Enter/Exit logging (see 10.2.3) of functions by indenting the text between entering and exiting a function, e.g. the column TEXT displays: Enter FunctionXYZ() This is log statement 1. This is log statement 2. Exit FunctionXYZ() whereas the column FMT_TEXT displays: Enter FunctionXYZ() This is log statement 1. This is log statement 2. Exit FunctionXYZ() PROF_FUNC Defines: USE_PROFILING_COLUMNS=1 and USE_ISO_TIMESTAMP_COLUMN=1 Source file: “extendedcolumns.cdf” Time in milliseconds between entering and leaving a function based on the timestamps of the log records. This is useful for investigating the performance of a program. PROF_LINE Defines: USE_PROFILING_COLUMNS=1 and USE_ISO_TIMESTAMP_COLUMN=1 Source file: “extendedcolumns.cdf” Time in milliseconds between two subsequent log records based on the timestamps of the log records. This is useful for investigating the performance of a program. By setting the define FILE_SPECIFIC_LINE_PROFILING to 1 the column PROF_LINE will display the time between two subsequent log records originating from the same file. If you set this define to any other value than 1 the file origin is not considered. PROF_FUNC_ENTER Defines: USE_PROFILING_COLUMNS=1, 102/147 USE_ISO_TIMESTAMP_COLUMN=1 and USE_FUNCTION_ENTER_COLUMN=1 Source file: “extendedcolumns.cdf” When in a log record (row) one of the “enter function”-keywords is detected this column will contain the name of the function in this row. PROF_FUNC_EXIT Defines: USE_PROFILING_COLUMNS=1, USE_ISO_TIMESTAMP_COLUMN=1 and USE_FUNCTION_EXIT_COLUMN=1 Source file: “extendedcolumns.cdf” When in a log record (row) one of the “exit function”-keywords is detected this column will contain the name of the function in this row. PROF_FUNC_ENTER_EXIT Defines: USE_PROFILING_COLUMNS=1, USE_ISO_TIMESTAMP_COLUMN=1 and USE_FUNCTION_ENTER_EXIT_COLUMN=1 Source file: “extendedcolumns.cdf” When in a log record (row) one of the “enter function” or “exit function”-keywords is detected this column will contain the name of the function in this row, e.g. FunctionXYZ in the example given for FMT_TEXT (see Table 10.2) ISO_TIMESTAMP Defines: USE_ISO_TIMESTAMP_COLUMN=1 Source file: “extendedcolumns.cdf” A special timestamp format YYYY-MM-DD hh:mm:ss,mmm is required for some features of Ascolog Insight (e.g. for creating graphs). ISO stands for International Organization for Standardization. If the timestamps of your log records have a different format you can convert them to this ISO timestamp. The converted timestamps will be provided in the ISO_TIMESTAMP column. You must specify in the Options dialog in which column (category Columns, field TimestampColumn) the conversion command can find the timestamps that should be converted. You also have to specify the date format used by these timestamps (category Calculation, field SourceTimestampFormat) because Ascolog Insight cannot determine the order of year, month and day in a date (e.g. 12-09-11, is 11 or 12 the year?). Table 10.2: Predefined custom columns 10.2.2 Performance Analysis of A Program The PROF_FUNC and PROF_LINE are two predefined custom columns which are very helpful when you have to analyze the performance of a program. For a detailed description see 10.2.1 Custom Columns Overview, Table 10.2. 103/147 10.2.3 Visualizing Call Sequences of Functions -Enter/Exit Logging The term function in this section refers to functions or methods from programming languages and not to analyze functions used by Ascolog Insight. Ascolog Insight provides some useful methods for visualizing call sequences of functions: • Visualization in a graph using alternating lines • Predefined custom columns You can only use these methods when the log records contain keywords which can be used to identify the beginning and the end of a function call. These keywords must be located at the beginning of the log record's description part (the description is represented by the TEXT column). The keywords which are supported by Ascolog Insight are • Enter, Entering, Enter Function, Exit Function • Leave, Leaving, Leave Function, Leaving Function • Exit, Exit Function The recognition of these keywords is case-insensitive. Example 10.1 shows a log which uses the keyword “ENTER” and “EXIT”. 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 2013-02-27 23:11:02,675 23:11:04,703 23:11:04,703 23:11:04,883 23:11:09,911 23:11:22,102 23:11:22,104 23:11:22,104 23:11:22,104 23:11:27,756 23:11:27,763 23:11:27,764 23:11:27,813 23:12:29,380 23:12:31,313 23:12:31,315 23:12:31,463 23:12:42,354 23:12:42,378 23:25:17,691 23:25:19,639 23:25:19,640 23:25:19,785 23:25:42,163 23:25:42,164 23:25:42,165 23:25:42,186 I I I I I I I I I I I I I I I I I I I I I I I I I I I 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 0x1D50 ENTER Function1 ENTER Function2 LogLine EXIT Function2 ENTER Function3 LogLine LogLine ENTER Function4 LogLine LogLine EXIT Function4 LogLine LogLine LogLine LogLine EXIT Function3 LogLine LogLine LogLine ENTER Function2 LogLine LogLine LogLine LogLine LogLine EXIT Function2 EXIT Function1 Example 10.1: A log using the keywords "ENTER" and "EXIT" The predefined columns (FMT_TEXT, PROF_FUNC and PROF_FUNC_ENTER_EXIT) are described in chapter 10 Custom Columns Definitions (CCD), Table 5.3 Filter expressions. 104/147 About FMT_TEXT and PROF_* columns and grammars When creating a grammar it is very IMPORTANT to assign any whitespace between the header and the description of a log record to the header, otherwise formatting and profiling of call sequences of functions won't work correctly. When such whitespace is not part of the header it becomes part of the FMT_TEXT column, i.e. the FMT_TEXT column starts with whitespace. However, the FMT_TEXT must start with one of the keywords allowed for enter/exit logging and not with a whitespace. In the Graph dialog there is the possibility to set the Draw mode for lines to alternating. Alternating lines are intended to visualize call sequences of functions. The idea is that a line is drawn during the execution of the function, i.e. the line starts when in a computer program a function is entered and it ends when the function is left. Let's look at an example to understand why it is called alternating lines. When the log records A, B, C, D, E, F are drawn in a graph and alternating lines is selected in the graph options, Ascolog Insight will draw a line between A and B, C and D, E and F. No lines will be drawn between B and C, D and E. In order to draw a call sequence of a function you have to set the Draw mode to one of the alternating styles but you also have to create a graph which draws a graphical node for all log records logging the entering and the exit of this function. For the function with the name Function2 from Example 10.1 possible filter expressions for the graph are shown in Example 10.2 and Example 10.3. You can check if the log record contains the function name in the FULLROW column as it is done in Example 10.2. FULLROW ~= "Function2" Example 10.2: Filter expression for Function2 using the FULLROW column Alternatively, you can use the predefined custom column PROF_FUNC_ENTER_EXIT which only contains the function name (and nothing else) and check if the column contains the function name as shown in Example 10.2. PROF_FUNC_ENTER_EXIT == “Function2” Example 10.3: Filter expression for Function2 using the PROF_FUNC_ENTER_EXIT column Each function has to be displayed on its own timeline, it is not useful to display two or more functions on the same timeline, i.e. for each function you have to create an own graph and an own filter expression. Hint: Use the Group command for creating all these graphs in a single step. Right-click on the PROF_FUNC_ENTER_EXIT column and select Group/Graph/Timeline/Auto from the context menu. If in a log file each entering of a function is followed by an exit the Draw mode Alternating will have the effect that a line is drawn between the graphical node representing the entering of a function and the graphical node representing the exit of a function. If in the log files there is a log record that logs the exit of a function before a record logs an entering you have to filter out this “exit” log record, otherwise lines will be drawn when the 105/147 program is not executing the function which is wrong. A filter to correct this where “2013-10-16 14:35:49,951” is the timestamp of the “exiting” log record could look like in Example 10.4. (FULLROW ~= "Function2" && !( ISO_TIMESTAMP == "2013-10-16 14:35:49,951" )) Please note that the exclamation mark in the expression filter represents a logical NOT. Example 10.4: Filtering out the first "exit" log record If you want to use alternating lines for multi-threaded applications you have to create an expression filter that can distinguish between the different threads, i.e. the information what thread has logged a log record has to be part of the log record. 10.3 Controlling Custom Columns using a Graphical User Interface Ascolog Insight offers the possibility to control custom columns by using a dialog called CCD Control that manipulates the parameters of the command/macro which creates the custom column. In the CCD Data dialog you can assign command/macro parameters to controls of the CCD Control dialog. Both dialogs can be opened via the Tools menu where you can find the CCD Data... and CCD Control... commands. CCD is an abbreviation for custom columns definition. A use case of this feature is time shifting. The next section will demonstrate with this use case how custom columns can be controlled using the CCD Control dialog. By studying this example you will be able to use this feature for other use cases. 10.3.1 Correcting Timestamps (Time Shifting) The intention is to merge two or more log files by using their timestamps as merge criteria. The order of the log records in the merged log can be incorrect if the timestamps were created by different clocks that were not synchronous. Ascolog Insight offers you the possibility to correct timestamps by the same but configurable value so that the order of the records in the merged log can be corrected. This is called time shifting. In order to enable time shifting the following steps are necessary: (1) The TIME_SHIFT macro which implements the time shift functionality has to be added to the layout's custom columns definition by using the Custom Column Definition editor. (2) Time shifting requires three parameters. They have to be configured in the CCD Data dialog. Parameter ts_cor_mul_1: Corrects the unit of the parameter ts_cor_1. Parameter ts_cor_1: The timestamps will be shifted by the value of this parameter. The unit (seconds, minutes, etc.) of this parameter depends on the value of ts_cor_mul_1 because the TIME_SHIFT macro multiplies ts_cor_1 with ts_cor_mul_1 and considers the product of these two values to be a time in milliseconds. If you want to set the unit of ts_cor_1 to minutes you have to set ts_cor_mul_1 to 60000 because 1 minute * 60000 = 60000 milliseconds. Parameter ts_fn_1: Time shifting will be applied to the log file specified by this parameter. 106/147 The first step is to add the TIME_SHIFT macro to the custom columns definition. Go to the Tools menu and open the Custom Column Definition dialog. Select Edit and Defines... to open the Defines dialog. Add USE_ALL_MACROS=1 in the Defines dialog to enable the TIME_SHIFT macro. Following this, go to the Tools menu and select Includes... to open the Includes dialog. Add the folder where the file is located that contains the TIME_SHIFT macro (file “basemacros.cdf”) to the custom columns definition's include directories. How to find the file “basemacros.cdf” is explained in Figure 10.3 in section 10.1 Working with Custom Columns Definitions. Moreover, the file “basemacros.cdf” has to be included in the custom columns definition using the statement #include “basemacros.cdf” and the TIME_SHIFT macro has to be added, too. The custom columns definition has to look like in Example 10.5. The macro gets a timestamp column which is created by the layout's grammar definition as input (TIMESTAMP in the example). It shifts all values of the log file which is specified in the CCD Control dialog by the shift value set in the CCD Control dialog. In the example the output is passed to the TS column. #include "basecolumns.cdf" #include "basemacros.cdf" // this custom column will serve as an adjusted timestamp TS = TIME_SHIFT(TIMESTAMP); Example 10.5: Shifting timestamps Press Save & Apply in the File menu of the Custom Column Definition dialog to save the changes made and to apply the custom column definition to the currently loaded log records. The Custom Columns Definition dialog will be closed. The TIME_SHIFT macro adds the TS column to the Log window. The next step is to open the Options dialog which is located in the Tools menu. Set the Columns/MergeColumn property to the TS column. Since the TS column will contain the time shifted timestamps it is useful to set TS as the MergeColumn to use the time shifted timestamps for timestamp comparisons. Sort the TS column in ascending order. In order to configure the parameters mentioned in step 2 open the CCD Data dialog which is shown in Figure 10.6. Press the Add button to add a new parameter. Select a parameter and press Edit to edit a parameter. When you press Remove the selected parameter is deleted whereas Clear deletes all parameters. 107/147 Figure 10.6: Defining parameters for custom columns commands After adding all parameters the dialog should look like in Figure 10.6. The following description will guide you through this process: First add the parameter ts_cor_1. The parameter ts_cor_1 should be configured as shown in Figure 10.7. Assign the Slider control 1 to the parameter. Using this control you can specify the time value that is used to shift timestamps. The slider control will be used in the CCD Controls dialog to control the parameter. The variable %Slider1 will contain the value set by the Slider control 1. The value of %Slider1 will be assigned to the parameter ts_cor_1. The Min and Max fields specify that the time shift values can be set from -3600 to 3600. The initial value (Value field) will be zero so timestamps are not shifted. 108/147 Figure 10.7: Editing the ts_cor_1 parameter The parameter ts_cor_mul_1 must be configured as shown in Figure 10.8. This parameter just corrects the unit used for time shifting which is why no control is assigned. Since ts_cor_mul_1 is set to 60000 the time shift values of ts_cor_1 are minutes. Thus you can shift timestamps for 3600 minutes back and forth (see option Slider control default setting of Figure 10.7). Figure 10.8: Editing the ts_cor_mul_1 parameter There is still another parameter to configure. An edit control of the CCD Controls dialog should be assigned to the parameter ts_fn_1 as shown in Figure 10.9. Using this control you can specify the log file whose timestamps should be shifted. The value of %Text1 will be passed to the parameter ts_fn_1. 109/147 Figure 10.9: Editing the ts_fn_1 parameter Click on the CCD Controls button in the toolbar to open the dialog shown in Figure 10.10 where the actual values for the time shift can be set. The time shifting functionality is now ready to use. Figure 10.10: Setting the values for the time shift In the Control set selection drop down list select 1. Control Set [ts_cor_1, ts_fn_1] because in the CCD Data dialog the first controls (Slider control 1 and Text control 1) were assigned to the ts_cor_1 and ts_fn_1 parameters. The TIME_SHIFT macro will apply the time shift on the file specified in the Text1 edit control because this edit control is assigned to the ts_fn_1 parameter. Tick the Merge records check box to tell Ascolog Insight to merge the time shifted log records 110/147 with the other log records. This means that the log records from the different files will be ordered by their timestamps in the Log window. A time shift will always be applied to the original timestamp as it is logged in the log file. When you enter -60 in the Slider1 edit control and press the Apply button the timestamps of the log records from file “Samplefile5.log” will be shifted back for 60 minutes, i.e. if the hour part of an original timestamp is 10 the shifted timestamp's hour part will be 09. If you change the time shift value from -60 to 60 minutes this will mean that the hour part of the shifted timestamp will be 11. because the original hour timestamp value (=10) will be used and not the time shifted one (=09). Instead of directly specifying the time shift value in the edit control and pressing the Apply button you can also use the slider. Changes to the slider will be immediately applied. You can also use the "Arrow-left" and "Arrow-right" keys to immediately shift the timestamps. 111/147 11 Merging Logs -Advanced Topics 11.1 How Merged Logs are Created In order to create a merged log the application retrieves the first log record from each source log and compares them. The log record that has the smallest value in the merge column (e.g. the oldest timestamp) is the next record to be copied to the merged log. For more information about the merge column can be found in 4.5 Loading, Merging, Browsing and Closing Logs (!). A copied record is not considered anymore in the next comparisons but its succeeding log record. The merging algorithm terminates after all records have been copied to the merged log. It might occur that the timestamps of log records within a log are not ordered. This is shown in Example 11.1: 2011-10-10, 2011-10-10, 2011-10-10, 2011-10-10, 15:20, 13:00, 13:01, 15:22, Log Log Log Log record record record record 1 2 3 4 Example 11.1: The timestamps are not ordered This might happen when someone changed the system clock between the creation of these log records: It can be assumed that “Log record 2” and “Log record 3” were created after “Log record 1” although they have an older timestamp because they were written to the log after “Log record 1” and logs are usually created from top to bottom. However, the application can be configured to sort the log records within a log by their merge column before merging them with other log records or to leave them in the order they occur. Open the Tools menu and select Options. Expand the Performance category. If you set OmitMergeColumnSort to FALSE the merge column of a log will be sorted before merging, if you set this field to TRUE sorting will be omitted. 11.2 Changing the Merge Order When log records from different log files are merged they can have the same timestamp values. You can tell Ascolog Insight how to order these log records in the merged log by using the commands Merge Order Up and Merge Order Down from the Tools menu. Example 11.2 shows a merged log with log records from two different log files (“File1” and “File2”) which have the same timestamp values. 2011-10-10, 2011-10-10, 2011-10-10, 2011-10-10, 15:00, 15:00, 15:00, 15:00, Log Log Log Log record record record record 1, 2, 3, 4, File1 File1 File2 File2 Example 11.2: Log records from different files with the same timestamp value 112/147 If you want to change the order of these log records, select a log record of “File1” and execute the Merge Order Down command. Alternatively, you can select a log record of “File2” and execute the Merge Order Up command. As a result the log records will be ordered as shown in Example 11.3. 2011-10-10, 2011-10-10, 2011-10-10, 2011-10-10, 15:00, 15:00, 15:00, 15:00, Log Log Log Log record record record record 3, 4, 1, 2, File2 File2 File1 File1 Example 11.3: Changing the order of log records with Merge Order Down or Merge Order Up If the commands cannot be applied they are deactivated (displayed in gray). 11.3 Merging Records with Custom Columns 11.3.1 Merge Records and Recalc Records When Ascolog Insight merges several logs that contain columns created by a custom columns definition the following procedure is followed: 1. The values for the custom columns are calculated for each single log independently from the other logs before the logs are merged. This means that in the merged log the custom columns have their original value. This step 1 is called merge records and this step is always done when merging logs. 2. For the merged log created in step one the values for the custom columns are recalculated. The merged log is handled as if it is one single log. This means that the values of the custom columns depend on all input logs contributing to the merged log. This step is called recalc records and is optional. In the Tools menu and in the Toolbar you can find the menu items Merge Records and Recalc Records which you can use to switch between the merge records and recalc records mode. The menu item that is active is displayed in gray and cannot be selected again. If Recalc Records is always gray and selecting Merge Records does not have any effect it is because of the option AutoRecalcMode which can be configured in the Options dialog. For a more detailed description of AutoRecalcMode see section 11.3.2 Options. When you select the command Force Recalc Records from the Tools menu Ascolog Insight will execute the recalc records process. However, the current mode of merging custom columns will not be changed. In order to illustrate merge records and recalc records let's have a look at two log files which should be merged. Example 11.4 and Example 11.5 show how these log files are displayed in their file tabs in the Log window. In the custom columns definition the lnr command is used to create a custom column LINENUMBER which displays the line number of a log record. 113/147 TIMESTAMP 2011-10-10 15:01 2011-10-10 15:10 TEXT Log record 1 Log record 2 LINENUMBER 1 2 Example 11.4: Log file 1 as displayed in its tab in the Log window TIMESTAMP 2011-10-10 13:02 2011-10-10 15:05 2011-10-10 15:06 TEXT Log record 3 Log record 4 Log record 5 LINENUMBER 1 2 3 Example 11.5: Log file 2 as displayed in its tab in the Log window The line numbers in the merged log are calculated differently depending on whether merge records or recalc records is the active mode. The merged log records are ordered by their timestamps. The merged log when merge records is the active mode is displayed in Example 11.6. The log records have the same line number as in their file tab, so many line numbers occur twice. TIMESTAMP 2011-10-10 2011-10-10 2011-10-10 2011-10-10 2011-10-10 15:01 13:02 15:05 15:06 15:10 TEXT Log record Log record Log record Log record Log record 1 3 4 5 2 LINENUMBER 1 1 2 3 2 Example 11.6: Merged log when merge records is the active mode The merged log when recalc records is the active mode is displayed in Example 11.7. The line numbers for the custom column LINENUMBER are recalculated so that each log record is assigned the line number of its position in the the merged log. TIMESTAMP 2011-10-10 2011-10-10 2011-10-10 2011-10-10 2011-10-10 15:01 13:02 15:05 15:06 15:10 TEXT Log record Log record Log record Log record Log record 1 3 4 5 2 LINENUMBER 1 2 3 4 5 Example 11.7: Merged log when recalc records is the active mode 11.3.2 Options You can also adapt merge records and recalc records to your needs. Open the Tools menu and select the Options command. In the category Calculation you can find the options for merge records and recalc records as they are shown in Table 11.1. Recalc options Description AutoRecalcMode Defines the recalculation behavior upon record merge: MANUAL: Depending on the menu command (Merge Records or Recalc Records) either a merge records or a recalc records is performed. DYNAMIC: A recalc records is performed only if it makes sense (more than one log file is loaded). 114/147 FORCE: A recalc records is always performed. RecalcReverse Reverse recalculation processing order. This is an advanced setting RecalcMultiPass Execute an additional recalculation pass. This is an advanced setting and you usually don't have to change it. and you usually don't have to change it. RecalcReverse2ndPass Reverse recalculation processing order of second pass. This is an DoubleBuffering Use an additional buffer upon recalculation. This is an advanced advanced setting and you usually don't have to change it. setting and you usually don't have to change it. Table 11.1: Recalc options 115/147 12 Layouts – Advanced Topics This chapter covers advanced layout topics. Most users will be able to work with Ascolog Insight effectively without reading this chapter. However, before you study this chapter please read 4 Working with Logs and Layouts which is important for understanding this chapter. 12.1 Layout Files The information belonging to a layout is stored in several text files. The default location for a layout in the file system is the “LAYOUTS” folder which is a sub-folder of the data folder. The location of the data folder depends on the installation method you used. Where you find the data folder is described in 2.3.1 Installing with XCOPY (!) respectively 2.3.2 Installing with the Setup (!). A layout with the name TestLayout consists at least of the following files: “TestLayout.xml”, “TestLayout.cdf”, “TestLayout.gdf” and “TestLayout.ini”. If Ascolog Insight asks you to specify a layout you have to provide the location of the XML (e.g. TestLayout.xml) file. This file is the main file which knows the paths to the other files of the layout. Table 12.1 contains a detailed list of the data which is stored in a layout. Data Description File Grammar definition, Paths to the other files of the layout. “TestLayout.xml” column definition and configuration paths Record definitions Ascolog Insight supports the conversion of log records “TestLayout.xml” to a different format (menu File/Convert...). These conversions can be stored and reused. Custom columns Custom columns definition data (CCD data, see menu definition (CCD) data Tools/CCD Data menu) is passed to the commands in “TestLayout.xml” a custom columns definition which use this data to modify custom columns. You can use a GUI to specify values for CCD data. Tabs When tabs are made persistent their data is stored in “TestLayout.xml” the layout. Options All options that belong to a layout are stored in the layout. Application wide options are stored in the “TestLayout.xml” “base.ini” file. Columns Properties of the columns of the Log window. “TestLayout.xml” Functions tree The layout stores the analyze functions of the “TestLayout.xml” Grammar definitions Production rules, terminals and nonterminals of a “TestLayout.gdf” Functions tree like record filters, graphs, colors, etc. 116/147 grammar which define the format of a log type. Nonterminals can be represented by a column in the (Grammar Definition File) Log window. Custom columns definitions Custom columns definition are created by the user in “TestLayout.cdf” logs for analysis. The extracted data is represented in Definition File) order to extract additional meaningful data from the its own column in the Log window. Macro parsers (Column Paths to macros, defines which control the behavior of “TestLayout.ini” macros. Pattern definitions Regular expressions, definitions of nonterminal symbols. Table 12.1: Data stored in a layout 117/147 “TestLayout.ini“ 13 Grammars -Advanced Topics This chapter covers advanced grammar topics. Most users are able to work with Ascolog Insight effectively without reading this chapter. However, before you study this chapter please read 4 Working with Logs and Layouts which is important for understanding this chapter. 13.1 How Grammars are used in Ascolog Insight In Ascolog Insight a grammar (formal grammar) is used to describe the structure of log records. A grammar consists of nonterminal symbols, terminal symbols, production rules and a start symbol. These elements of a grammar can be modified in the Grammar Definition window. Table 13.1 shows how a grammar's elements are used in Ascolog Insight to identify the pieces of information of log records. Terminology Explanation Usage in Ascolog Insight Terminal A terminal symbol is a symbol which symbols Terminals in Ascolog Insight could be cannot be replaced by other symbols. timestamps, numbers, alphanumeric characters, letters, colons, etc. Ascolog Insight uses regular expressions to define the terminal symbols. Alphabet An alphabet is the set of all terminal symbols of a formal grammar. Nonterminal symbols A nonterminal symbol is replaced by Example: The header of a log record can symbols by applying a production By applying production rules an abstract other nonterminal or terminal rule. Notation: Nonterminals are marked in this documentation by angle be represented by a nonterminal symbol. header is transformed to a “real” header (a word in formal grammar terminology): <Header>::=<Timestamp>;<Description> is transformed to brackets: <nonterminal> 2013-12-24 10:23:23,000;File error! Production A production rule describes how to Example: <Severity> ::= HIGH|MEDIUM| rule transform a nonterminal symbol. The LOW nonterminal symbol to be replaced is Explanation: The nonterminal <Severity> located on the left side of the character sequence “::=”. It is replaced by the sequence of symbols can be replaced by either the terminal HIGH, MEDIUM or LOW. on the right side of “::=”. Both Example: nonterminal and terminal symbols <LogRecordHeader>::=<Timestamp>;<Se left side of “::=” there is exactly one Explanation: A header of a log record are allowed on the right side. On the verity>|<Timestamp> nonterminal symbol in the grammars consists of a timestamp (nonterminal), 118/147 supported by Ascolog Insight. separated by a semi-colon (terminal) from a severity level (nonterminal) OR only of a timestamp. Word or string A word is a finite sequence of The words in Ascolog Insight are the valid characters. A word is created from an headers (the complete header, not only alphabet by applying production rules. parts!) for the log records of a log type. Example: The severity INFO is not a word but a terminal! Formal Language The set of all strings which can be created by applying the production All headers of a log belonging to a certain log type. rules of the formal grammar. “|” The vertical bar indicates a choice. The following statement defines that a header consists of a <Level> OR the terminal “STOP”:<Header>::=<Level>| STOP Epsilon Denotes the empty terminal. The The following statement makes the empty terminal is used for options. nonterminal <Level> optional: <Header>::=<Level>|epsilon Symbol Repetitions must be defined by using The nonterminal <Digit> should be repetitions recursions, i.e. the nonterminal allowed to occur several times: right side. <Digit><DigitSequence> symbol on the left occurs also on the <DigitSequence>::=<Digit>| Table 13.1: Ascolog Insight's usage of a grammar When Ascolog Insight applies a grammar on a log file it first does a lexical analysis, i.e. it splits the content of the log file into meaningful pieces of information (tokens) based on the terminal definitions of the grammar. The next step is the syntactic analysis (parsing) where the tokens and production rules of the grammar are used to create a parse tree. In this step Ascolog Insight checks if the log file is valid according to the grammar. When the log file is valid the tree is parsed in order to assign the pieces of information of the log file to columns. Ascolog Insight uses a context-free LL(1) grammar which means that the context of a nonterminal symbol is not considered by a production rule. The context of a non-terminal symbol are its preceding and succeeding symbols. Thus on the left side of the character sequence “::=” there must be only one nonterminal symbol. LL(1) means that the production rule which must be applied depends only on the first next token in the log file that was not processed yet. Ascolog Insight uses a notation similar to the Backus-Naur form to describe a grammar. The Backus-Naur form is a common notation for context-sensitive grammars. In Ascolog Insight a header must be located after a carriage return or at the beginning of the log. If all log records are identified the structure of the log is known. If no header is found at the beginning of a log an error message is displayed. In order to identify a header regular expressions would also be sufficient in many cases. However, regular expressions cannot be used to identify the single parts of a header which is needed to effectively analyze the content of a header. 119/147 13.2 Grammar Definition Dialog 13.2.1 An Introduction to the Grammar Definition Dialog Figure 13.1: Grammar Definition dialog Figure 13.1 shows the Grammar Definition dialog. In the title bar of the Grammar Definition dialog Ascolog Insight displays the currently loaded grammar definition file. Open the Help menu and select Info to learn more about the commands of the Grammar Definition dialog's menu. In the File menu you find useful commands for working with the grammar definition file. Table 13.2 lists these commands. File menu Description New Creates a new grammar which treats log files as plain text, i.e. no headers are identified and each line is considered to be a log record. Therefore Ascolog 120/147 Insight adds the sof and eof terminal symbols and for the header a nonterminal symbol which is empty. Load Loads a grammar definition file (file extension: “.gdf”). A previously loaded grammar definition file will be closed and any changes that were not stored will be discarded. Merge/File Merges a grammar definition from a grammar definition file with the current terminal and nonterminal symbols. Terminals: If a terminal symbol with the same name exists in both grammars and if it has the same definition in both grammars this terminal symbol will only occur once in the merged grammar. If a terminal symbol with the same name exists in both grammars but the definitions are different both terminal symbols will be copied to the merged grammar. You will have to decide what definition you want to keep. Nonterminals: How nonterminal symbols with the same name are handled is defined in the Options menu by the option Nonterminals/Progressive Merge. Merge/Top This command is similar to Merge/File but one of the sources won't be a grammar definition file but the topmost grammar from the grammar stack. Use the Push and Pop commands of the File menu to manipulate the grammar stack. Save & Apply Saves the current terminals and nonterminals to the grammar definition file displayed in the title bar and immediately applies this grammar. Save As Saves the current terminals and nonterminals to a grammar definition file. The Save Grammar Definition File dialog will be opened where you have to provide the location and the name for the grammar definition file. Push Stores the current grammar definition (terminals, nonterminals and production rules which are currently displayed in the dialog) to the grammar stack. You can put several grammar definitions to the grammar stack. Pop Fetches the topmost grammar definition from the grammar stack and replaces Exit Closes the Grammar Definition dialog without saving the grammar definition. the current grammar definition with this topmost grammar definition. Table 13.2: Grammar Definition dialog/File menu The commands of the Terminals menu are described in Table 13.3. Options/Terminals menu Description Lock This option is also available as check box directly above the Terminals list. If you provide a text sample and press the Scan button Ascolog Insight will create the necessary terminal symbols. Terminal symbols that have 121/147 already existed before the scan was triggered are kept when Lock is checked. Otherwise they are removed. Include Predefined When you provide a text sample and press the Scan button Ascolog Insight will create a grammar definition. If you tick this option it will consider the predefined terminals (e.g. for timestamps, alphanumeric character sequences, etc.), when creating the grammar. For most use cases enabling this option makes perfectly sense. A reason for not enabling this option is that you want to use your own terminal definitions which are not compatible with the predefined terminal definitions. Remove Unreferenced Removes unreferenced terminals (displayed in light gray) from the Terminals list. Unreferenced terminals are terminals which do not occur in a production rule or macros which do not occur in a terminal definition. Add Predefined Adds predefined terminals to the Terminals list. Predefined terminals are available for timestamps, spaces, paths, file names or alphanumeric character sequences. Table 13.3: Grammar Definition dialog/Options/Terminals menu The commands of the Nonterminals menu are described in Table 13.4. Options/Nonterminals menu Description Progressive Merge Nonterminal symbols that exist in grammars which shall be merged will be combined using the logical OR (the OR is represented by “|”): Merging HEADER ::=NONTERMINAL_1 and HEADER::=NONTERMINAL_2 will yield HEADER::=NONTERMINAL_1 | NONTERMINAL_2 Default Postfix Options When you provide a text sample and press the Scan button Ascolog Insight will usually create several nonterminal symbols. The names will start with “NT” which stands for NonTerminal. The following options define how Ascolog Insight will create postfixes to generate distinct names for the nonterminal symbols. Default Postfix > Number An increasing number will be used for the end 122/147 of the nonterminal's name, e.g. NT_1, NT_2, NT_3, etc. Default Postfix > Hash A hash value will be used for the end of the nonterminal's name, e.g. NT_19A96BEA, NT_19A96BEB, etc. Default Postfix > GUID A GUID (Global Unique Identifier) will be used for the end of the nonterminal's name, e.g. NT_F9CBB411-1DCB-4077-A535EB6B221AE9EE, NT_F9CBB411-1DCB-4077-A535EB6B221AE9EE, etc. Remove Unreferenced Remove unreferenced nonterminals (displayed in light gray) from the Nonterminals list. Unreferenced nonterminals are nonterminals which do not occur in a production rule. Table 13.4: Grammar Definition dialog/Options/Nonterminals menu Terminal symbols, nonterminal symbols and production rules are case-sensitive. On the right side of the Terminals and Nonterminals list (see Figure 13.1) you can find the commands which allow you to modify these lists. The commands are described in Table 13.5. Command Description Add Adds a terminal or nonterminal symbol to the Insert Inserts a terminal or nonterminal symbol before end of the list. the selected symbol. Edit Starts the Edit dialog of the selected Clear Removes all terminals or nonterminals from the nonterminal or terminal symbol. list. Remove Removes the terminal or nonterminal that is Arrow up Moves the selected terminal or nonterminal one selected from the list. position up in the list. Arrow down Moves the selected terminal or nonterminal one position down in the list. Table 13.5: Manipulating grammar definitions In the Status line of the Grammar definition dialog Ascolog Insight displays information about the status of the grammar and the text sample if provided. When everything is correct the text Valid is 123/147 displayed, otherwise an error message is displayed. The most important messages with their description are displayed in Table 13.6. Message Status Valid The provided text sample is valid according to Incomplete – Nonterminal: <name of the first The provided parts of the text sample are nonterminal symbol which is missing> correct but some parts are missing. the grammar definition. This message is also displayed when no text sample is provided. Text mismatch <additional information> This message is displayed when the provided text sample does not match the grammar. Depending on the error additional information is displayed. Invalid first-set A grammar is defined that is not of type LL(1). Table 13.6: Most important grammar status messages 13.2.2 Terminals The Terminal list of the Grammar Definition dialog contains all terminal symbols which belong to the current grammar. When you press the Clear button all terminal symbols will be removed. Pressing the Remove button only deletes the currently selected terminal symbol. Press the Arrow Up or Arrow Down button to move the selected terminal symbol one row up or down in the list. When you press the Add, Insert or Edit button of the Terminals list the Layout Definition Terminal dialog shown in Figure 13.2 will open. If you press the Edit button the dialog will display the current settings of the selected terminal symbol otherwise the dialog's fields will be blank. Figure 13.2: Defining a terminal In the Name field enter the name of the terminal symbol. In the Definitions field you must define the terminal symbol. Regular expressions are used to define a terminal. Note: Spaces in terminal 124/147 definitions are evaluated as any other characters (e.g. “A” or “9”). The supported regular expression operators are listed in Table 13.7. Operator Description Example () Group (abc) * Zero or more occurrences (abc)* + One or more occurrences (abc)+ ? Zero or one occurrence (abc)? #n n occurrences (abc)#10 | Alternative abc|def ~ Invert ~(abc) [] Character set [abc] [^] Inverted character set [^abc] [n-m] Character range set [a-z] \ Escape \* %xx ASCII code (hexadecimal) %0d%0a . Wildcard .+ ^ Start of field ^abc (column in filter expressions, text in $ End of field abc$ log) Table 13.7: Supported operators in regular expressions The Layout Definition Terminal dialog also has another useful feature which is called Update grammar dependencies. If this check box is ticked and you change the name of the terminal all occurrences of this terminal (e.g. in production rules) will also be changed to the new name. Besides name and definition you must also set the type of the terminal. Table 13.8 lists and describes all available types. Type Icon Description Macro Gray diamond A macro assigns a name to a regular expression in order to facilitate reusing this regular expression in other terminal definitions. This makes terminal definitions easier to read. Separator A single red leaf node Separators separate different pieces of information. from a tree graph Terminals of type separator are only relevant during the space symbol during parsing (column creation) because they are meta underlined by a white lexical analysis (scanning), they are not evaluated Evaluate information. A single red leaf node The information that is represented by this terminal is from a tree graph evaluated during parsing and will be displayed in a column. 125/147 Skip A single white leaf The information that is represented by this terminal is graph should only use this for special cases like the Start of node from a tree skipped and thus will not be displayed in a column. You File (sof) terminal. The begin of the log is represented by the terminal sof. This terminal should be skipped if no production rule should depend on this terminal. Otherwise it must be considered in a production rule in order to create a useful grammar. Skipped terminals will be ignored during parsing. Table 13.8: Terminal types Figure 13.3 shows the usage of macros, terminal types and nonterminal types in the Grammar Definition dialog. Figure 13.3: Notations used by the Grammar Definition dialog 126/147 In order to use a macro in a terminal definition you must encompass the macro with angle brackets (see dateTime terminal in the figure below). Moreover, when a macro should be used in a terminal definition you must tick the check box Resolve Macros in the Layout Definition Terminal dialog, i.e. in the grammar definition of Figure 13.3 you have to do this for the dateTime terminal symbol. Otherwise the angle brackets and the macro name are considered as “normal” characters. The definition of the dateTime terminal symbol mixes a macro and regular expressions in order to illustrate the usage of macros. It certainly makes sense to replace all other regular expressions [09] with the macro. The colon in Figure 13.3 in the HEADER nonterminal symbol before the nonterminal TID is the terminal “:” from the Name column of the Terminals list. The colon in the dateTime terminal's definition does not have anything to do with the terminal “:”. Terminals of type Separator or Evaluate cannot be used in the Terminals list. If you tick the check box Update Grammar Dependencies in the Layout Definition Terminal dialog of the terminal “:” and you change the name “:” to “colon” than in the HEADER nonterminal symbol the “:” will also change to “colon” (the notation is then analogous to “space”). 13.2.2.1 Ambiguous Regular Expressions The regular expressions of the grammar's terminal symbols must not be ambiguous, i.e. there must not be two regular expressions in a grammar definition that match the same character sequence. Examples for ambiguous regular expressions are given in Table 13.9. Regular expression Match [A-Za-z0-9] and [0-9] Both expressions match a character sequence consisting only of natural numbers: 123, 0333, 98 [A-Za-z0-9] and ERROR | WARNING | INFO Both expressions match the character sequences ERROR, WARNING, INFO and nothing else. [~(0-9)] and [A-Z] Both expressions match a character sequence consisting of capital letters of the alphabet, e.g. ERROR, LOG, FOLDER but not Line2, A&O Table 13.9: Ambiguous regular expressions Ambiguous regular expressions lead to problems when Ascolog Insight parses a log. Ascolog Insight displays an error message when it encounters an ambiguous regular expression. If an ambiguous regular expression is detected the one that better describes the character sequences which can occur in a log file should be kept. It is not necessarily a disadvantage when regular expressions are used that are quite general, e.g. [A-Za-z] instead of [A-Z]. 127/147 13.2.2.2 The Problem with Ambiguous Regular Expressions When Ascolog Insight analyzes a log the first step is the lexical analysis. In the lexical analysis the goal is to assign character sequences of the log to terminals. If terminals with ambiguous regular expressions are allowed (e.g. [0-9] and [A-Za-z0-9]) the application cannot decide what terminal better matches a character sequence because Ascolog Insight does not know anything about the meaning of the provided information. The log records in Example 13.1 will be used to illustrate the problem with ambiguous regular expressions. 2011-09-30 23:59:14 1234 Setup started. 2011-09-30 23:59:16 AB12 Setup did not complete. Example 13.1: In these two log records a hexadecimal code for the log record type is provided after the timestamp The character sequence 1234 from the first log record is both numeric and alphanumeric. If terminals with ambiguous regular expressions are used (e.g. [0-9] and [A-Za-z0-9] the application cannot decide what terminal better matches 1234 because Ascolog Insight does not know the meaning of the provided information (that 1234 is a hexadecimal code). In the next log record the character sequence AB12 occurs which is alphanumeric but not numeric. In order to see the subsequent problems let us assume that Ascolog Insight identifies 1234 as a numeric terminal and AB12 as an alphanumeric terminal. In order to identify the headers of log records Ascolog Insight will have to use production rules in the syntactic analysis which is done after the lexical analysis. Since 1234 was identified as a numeric terminal the production rule <LogRecordType> ::= alphanumericTerminal will not work. As a consequence, this production rule must be replaced by <LogRecordType> ::= alphanumericTerminal | numericTerminal , otherwise the header cannot be correctly identified. This illustrates that ambiguous regular expressions require more complex production rules. 13.2.3 Nonterminals and Production Rules In the Grammar Definition dialog there is also a list labeled Nonterminals. This list contains the nonterminal symbols and production rules of the grammar. The production rules define how to transform a single non-terminal symbol (Name column on the left) to one or more non-terminal or terminal symbols (Production column on the right). The process of identifying the pieces of information of a log and building a data structure (parse tree) is called syntactic analysis or parsing. During parsing the correct syntax of the log is checked. The result of the syntactic analysis is a parse tree. The syntactic analysis is based on the grammar's production rules. When you press the Clear button all nonterminal symbols will be removed. Pressing the Remove button only deletes the currently selected nonterminal symbol. Press the Arrow Up or Arrow Down button to move the selected nonterminal symbol one position up or down in the list. 128/147 When you press the Add, Insert or Edit button of the Nonterminals list the Layout Definition Nonterminal dialog shown in Figure 13.4 will open. In the case you pressed the Edit button the dialog will display the current settings of the selected nonterminal symbol otherwise the fields will be blank. Figure 13.4: Layout Definition Nonterminal dialog In the Name field enter the name of the nonterminal symbol. In the Definitions field you must define the production rule which describes how to process the nonterminal symbol. If you tick the Optional check box the nonterminal symbol will be optional. Making a nonterminal symbol optional means appending the OR operator “|” and the empty terminal “epsilon”. In production rules spaces are used to separate the terminal and nonterminal elements (like in Figure 13.4 between alphanum and “|”). Spaces also improve the readability of a production rule for humans. If you actually want to use spaces in production rules that are part of the formal language and match spaces from the log file you must define a terminal of type separator that defines spaces and use this terminal in the production rule (see Figure 13.5, space terminal symbol). The Layout Definition Nonterminal dialog also has another useful feature which is called Update grammar dependencies. If this check box is ticked and you change the name of the nonterminal all occurrences of this nonterminal (e.g. in production rules) will also be changed to the new name. Type Icon Description Evaluate A tree graph symbol with a yellow root The information that is represented by node and two red leaves nodes this nonterminal is evaluated and will be displayed in its own column in the Log window. Skip A tree symbol with a white root node The information that is represented by and two white leaf nodes this nonterminal is skipped and thus cannot be displayed in a column of the Log window. Table 13.10: Nonterminal types 129/147 Besides name and definition you must also set the type of the nonterminal symbol. The available types are described in Table 13.10. Figure 13.5 shows the use of nonterminals and production rules in the Grammar Definition dialog. Figure 13.5: Notations used by the Grammar Definition dialog The application only allows grammars that are of type LL(1). When a LL(1) grammar is used the parser only needs to look at the next token to make a parsing decision (decide what production rule has to be applied). If a user creates a grammar in the grammar definition window which is not of type LL(1) an error message is displayed. The production rules of Example 13.2 are not allowed in LL(1) and create the error message “Invalid first-set”. The reason is that the parser has to look at the second token of the next two tokens to make a parsing decision because the first token must be alphanumeric in both cases: 130/147 NONTERMINAL_1 ::= NONTERMINAL_2 terminal_a | NONTERMINAL_2 terminal_b NONTERMINAL_1 ::= alphanum NONTERMINAL_2 ::= alphanum Example 13.2: Invalid production rules in LL(1) 131/147 14 Exporting Logs 14.1 An Introduction to Exporting Logs Sometimes it is necessary to process logs outside of Ascolog Insight with other applications or tools which require a different log format. Therefore Ascolog Insight can convert logs to a different format and save the converted log in a file. It is even possible to extend Ascolog Insight using Ascolog Insight's Software Development Kit so that you can export logs to a database or process logs using syslog. Ascolog Insight supports the following conversion and exporting methods: 1. Converting the format of a log to a different format (changing the order of the information, removing some information) 2. Converting to a different character encoding (ANSI, UTF8, UTF16) 3. XML, HTML, HTMLREPORT 4. SENDMAIL, SYSLOG XML, HTML, HTMLREPORT, SENDMAIL and SYSLOG are plug-ins which are built with Ascolog Insight's SDK. Plug-ins are used to extend the functionality of Ascolog Insight. Please note that SENDMAIL and SYSLOG are only sample implementations. In order to extend Ascolog Insight you have to purchase a license for the software development kit of Ascolog Insight. If you require for your situation a special solution please contact us to learn more or to get a quote. Please have a look at 16 Ascolog Insight's Software Development Kit (SDK) for a brief overview of the SDK. All this functionality can be accessed via the Conversion dialog so first this dialog is introduced. After that the various conversion and exporting methods are explained in more detail. 14.2 Conversion Dialog Open the File menu and select Convert... to open the Conversion dialog. In this dialog you can define how to convert a log. This definition is called a conversion rule. A conversion rule can be made persistent and can be reused later. You can store a conversion rule by entering a name in the Persistent record definitions list control and by clicking on the Store button. To use an existing conversion rule select it from the Persistent record definitions list control and click Load. Pressing the Delete button will remove the selected conversion rule. The Columns list control displays all columns that are available in the currently loaded layout. If you want the information of a column to be part of the converted log you must add the column to the Record definition list. You can do this by selecting the column and by clicking Add to add it to the end of the Record definition or Insert to insert it before the column which is currently selected in the Record Definition list. Besides columns you can also add separators from the Separators list 132/147 (spaces, tabs, etc.) in the same manner as for columns. Next to the Record definition list control you have buttons to manipulate the record definition. Clear removes all items from the Record definition list whereas Remove removes only he currently selected item from the Record Definition list. Arrow up and Arrow down move the currently selected item up respectively down in the Record definition list. In the Options you can select the conversion/exporting method in the Encoding drop-down list: • ANSI, UTF8, UTF16 • XML, HTML, HTMLREPORT • SENDMAIL, SYSLOG In the File name edit control you must provide the destination of the log conversion. The destination can be a file but it can also be e.g. a syslog server or a database server. To use a server as the destination of the conversion you must provide a .ini file which must contain all required information to communicate with this server (e.g. IP address and port). You can click on the button on the right side of the File name edit control to open a file selection dialog. You can also specify whether the converted log records shall be appended or whether the existing log records shall be overwritten (check box Append data). In the Record filter edit control you can specify a filter so that only log records are converted that match this filter. Press the button with the dots (“...”) to start a dialog that supports you in creating a record filter. To read more on how to create record filters go to the section 5 Analyze Functions. When you press the Apply Active Record Filter button Ascolog Insight creates a record filter for the conversion based on the active record filters of the Functions tree. The created record filter will be displayed in the Record filter edit control. In order to execute a conversion/export press the Save button on the bottom of the dialog. When you press the Preview button you can see an excerpt of the input which is passed to the conversion/export. Please note that the preview DOES NOT show the result of the conversion/export, e.g. the content of the destination file. If you don't want to convert logs press the Cancel button. Be careful to store the conversion rule before pressing the Cancel button when you want to reuse it. 14.3 XML Ascolog Insight is delivered with a sample plug-in that converts logs to XML. Select the columns you want to convert and save in the XML file and add them to the Record definition list. Do not add separators including carriage returns (CR) and line feeds (LF) to the Record definition list. This will create XML files that consist of only one line and that do not make a lot of sense. When a record definition specifies that the data from the columns UID and TYPE shall be converted Ascolog Insight will create the XML tags <UID> and <TYPE> and the data of these columns will be encompassed by these tags as shown in Example 14.1. 133/147 <ASCOLOG_XML> <record><UID>423</UID><TYPE>3</TYPE></record> <record><UID>424</UID><TYPE>2</TYPE></record> more log records... </ASCOLOG_XML> Example 14.1: Exporting the columns UID and TYPE to XML 14.4 HTML The HTML plug-in is a sample implementation and is intended to be used as reference for implementing own HTML plug-ins. Ascolog Insight is delivered with a sample plug-in called HTML that converts logs to HTML. An HTML table is used to display the logs. The HTML plug-in is only a demo. It is based on Ascolog Insight's SDK. You can use the layout “TestLayout.xml” which is delivered with Ascolog Insight to study the HTML sample. Alternatively, add the code snippet listed in Example 14.2 to a custom columns definition of your layout. You can either put this code in an own custom columns definition file which you have to include in the layout's main custom columns definition or you can directly put the code into the layout's main custom columns definition file. #include <basemacros.cdf> #include <basecolumns.cdf> // columns used for HTML conversion #define _HTMLCOLUMN(name, label, img) fmt(label##"%#00"##img##".ico")) #define HTMLCOLUMN(name) #define HTMLESCAPE(txt) "&lt;", ">", "&gt;", "&", "&amp;") #define HTML_MAX_TEXT_LENGTH dti($HTML_column_##name, _HTMLCOLUMN(name, #@name, #@name) rpl(txt, "%0d%0a", "<br/>", "<", 80 HTMLINIT = ONCE(exe("HTMLInit", "", "Type", "Timestamp,left,10", "ThreadId", "ProcessId", "Text,left,30", "File,left,70")); HTML_TYPE = map(TYPE, TYPE, "0",HTMLCOLUMN(Trace), "1",HTMLCOLUMN(Info), "2",HTMLCOLUMN(Warning), "3",HTMLCOLUMN(Error), "4",HTMLCOLUMN(Fatal)); #if HTML_MAX_TEXT_LENGTH != 0 HTML_TEXT = HTMLESCAPE(cdt( { text == %1 }, dat($tmp, cpy(TEXT, 1, 0, HTML_MAX_TEXT_LENGTH)), dat($tmp), fmt("%s...", dat($tmp)))); #else HTML_TEXT = HTMLESCAPE(TEXT); #endif Example 14.2: Code snippet for exporting to HTML The command HTMLInit in the custom columns definition specifies the HTML table's header and the columns of the HTML table (e.g. width and text alignment). 134/147 Figure 14.1: Record definition for HTML conversion In order to export to HTML go to the Tools menu of the main menu and open the Options dialog. In the Calculation category set the AutoRecalcMode to DYNAMIC or FORCE. Then go to the File menu and select Convert... to open the Conversion dialog. In the Record definition list you must add all the columns which are passed as parameters to the command HTMLInit in the custom 135/147 columns definition in the same order, otherwise the HTML table won't be correct. Moreover, you have to set the Encoding to HTML and specify a file name in the File name field. The file name should have the extension “.html” or “.htm”. If your Conversion dialog looks similar to the one in Figure 14.1 press the Save button to convert the log records and save them to an HTML file. When you open the HTML file in a web browser you will see a table containing the log records similar to the one shown in Figure 14.2. If you do not see the icons in the Type column check if the paths to the icons in the created HTML file and the actual location of the icon files in the file system match. Figure 14.2: HTML table containing the log records 14.5 HTML Report The HTML Report plug-in is a sample implementation and is intended to be used as reference for implementing own HTML Report implementations. The HTML Report plug-in is another sample plug-in and very similar to the HTML plug-in. It generates an HTML file which consists of the Ascolog logo at the top , followed by the active Graph windows and at the bottom the log records are displayed in an HTML table in the same way as in the HTML plug-in (see Figure 14.2). 136/147 Figure 14.3: Record definition for HTMLReport conversion (setup installation) The approach is basically the same as for the HTML conversion so please also read 14.4 HTML. However, there are two exceptions (see Figure 14.3). In the Conversion dialog the Encoding field has to be set to HTMLREPORT. The File name field has to specify the path to the “report.ini” file which is located in the “PLUGIN_CFG” folder and provides the configuration settings for the plug137/147 in. The “PLUGIN_CFG” folder is a subfolder of the data folder and depends on the installation method that was used. How to locate the data folder is explained in 2.3.1 Installing with XCOPY (!) and 2.3.2 Installing with the Setup (!). The file name for the HTML output must be specified in the “report.ini” and NOT in the Conversion dialog. In the “report.ini” file there is a configuration setting called filename for this. The path is relative to Ascolog Insight's data folder. The default “report.ini” file offers the settings listed in Example 14.3. [HTMLREPORT] filename=.\temp\html\report.html logo=AscologLogo.png [GRAPH1] title=Graph1 ;width=0 ;height=0 ;start_time=2014-01-01 00:00:00,000 ;end_time=2015-01-01 00:00:00,000 ;y_min=0 ;y_max=100 [GRAPH2] title=Graph2 ;width=0 ;height=0 ;start_time=2014-01-01 00:00:00,000 ;end_time=2015-01-01 00:00:00,000 ;y_min=0 ;y_max=100 [GRAPH3] title=Graph3 ;width=0 ;height=0 ;start_time=2014-01-01 00:00:00,000 ;end_time=2015-01-01 00:00:00,000 ;y_min=0 ;y_max=100 [GRAPH4] title=Graph4 ;width=0 ;height=0 ;start_time=2014-01-01 00:00:00,000 ;end_time=2015-01-01 00:00:00,000 ;y_min=0 ;y_max=100 [TABLE] title=Table1 Example 14.3: Default content of "report.ini" 138/147 14.6 Syslog The syslog plug-in is a sample implementation and is intended to be used as reference for implementing own syslog implementations. Ascolog Insight is delivered with a sample plug-in called syslog. It is based on Ascolog Insight's SDK. Ascolog Insight can be used as syslog server and as syslog client. When Ascolog Insight is used as a syslog server it will create a thread which listens on a User Datagram Protocol (UDP) port. You can even run Ascolog Insight as a Windows® service. Purchasing an Ascolog Insight Professional license does not entitle you to use the Ascolog Insight SDK or to run Ascolog Insight as a Windows® service. If you are interested in this functionality please contact us to learn more or to get a quote. The sample demonstrates how syslog messages are sent from a syslog client to a syslog server. In the sample Ascolog Insight is started twice on the same computer and one instance is used as the syslog server whereas the other instance is the syslog client. The Ascolog Insight syslog client converts Microsoft Windows® event log records to the syslog format and sends them to the Ascolog Insight syslog server. Please consult a Microsoft Windows® or syslog documentation for further information. 14.6.1 Configuring the Syslog Server Start an instance of Ascolog Insight. This instance will be used as syslog server so you have to load the syslog layout (file “syslog.xml”) which is delivered with Ascolog Insight and which is located in the “LAYOUTS” folder. After loading the file the Files window displays the syslog configuration file (“syslog.ini”) including the complete path. The default location for the “syslog.ini” file is the “PLUGIN_CFG” folder that is part of the Ascolog Insight's data folder. The location of the data folder depends on the installation method that was used. How to locate the data folder is explained in 2.3.1 Installing with XCOPY (!) and 2.3.2 Installing with the Setup (!). The syslog server is now ready to be used because the syslog layout and the default “syslog.ini” already provide the appropriate settings for a localhost installation. Example 14.4 shows the default settings for the syslog server (section SYSLOG_RECEIVER). The server will listen on port 514 of the localhost and it will accept up to 100000 log records. [SYSLOG_RECEIVER] address = localhost:514 max_records = 100000 Example 14.4: "Syslog.ini" file which configures the receiver In the syslog layout the custom columns definition should already be correctly configured. More information about custom columns definition can be found in the chapter 10 Custom Columns Definitions (CCD). In order to check or correct these settings open the Custom Columns Definition editor and modify the layout's custom columns definition (menu Tools, command Custom Columns Definition...). In the editor go to the Edit menu and select Defines... to open the Defines dialog. By default the layout's defines should be configured as shown in Example 14.5. 139/147 USE_BASE_MACROS=1 USE_ALL_MACROS=0 USE_FULLROW_COLUMN=1 USE_UID_COLUMN=0 USE_NEWDATA_COLUMN=0 Example 14.5: Define settings You must also add the following custom columns definition files in the editor. The define settings above ensure that the required custom columns and commands are available in the included custom columns definition files shown in Example 14.6. #include <basemacros.cdf> #include <basecolumns.cdf> #include <syslog_basecolumns.cdf> Example 14.6: Required custom columns definition files Figure 14.4: Scheduled Tasks dialog Besides the configuration of the custom columns definition you also have to schedule the search and refresh task. In the syslog layout they should be already scheduled in the appropriate way as it is shown in the Scheduled Tasks dialog of Figure 14.4.However, if they are not you must schedule 140/147 them yourself. Open the Tools menu and configure the tasks as it is shown in the Scheduled Tasks dialog of Figure 14.4. The configuration of the refresh task is directly shown in Figure 14.4. The configuration of the search task is a little bit more complex. Select the search task and press the Edit button. This will open the Schedule Find in Files dialog. In this dialog you must specify the path to the “syslog.ini” file in the Look In field. The “.” refers to the data folder of Ascolog Insight. How to schedule tasks is described in detail in 9 Task Scheduling. If the scheduled tasks are correctly configured you must enable them if they are not enabled yet. Open the Tools menu and select Enable Scheduled Tasks. Task scheduling is enabled if the Enable Scheduled Task command is displayed in a light gray. If everything is correct the path to the syslog configuration file is displayed in the Files window and the status is Converted. 14.6.2 Configuring the Syslog Client Start another instance of Ascolog Insight. This instance will be used as syslog client which converts Microsoft Windows® event log records to the syslog format and sends them to the syslog server. You have to load the eventlog2syslog layout. The file “eventlog2syslog_service.xml” is located in the “eventlog” folder which is a sub-folder of the “LAYOUTS” folder. A text box as shown in Figure 14.5 might appear that informs you that the eventlog2syslog layout is not recommended for normal operation. You can confirm the dialog by pressing the OK button. Figure 14.5: Service layout warning In the eventlog2syslog layout the custom columns definition should already be correctly configured. More information about custom columns definition can be found in the chapter 10 Custom Columns Definitions (CCD). In order to check or correct these settings open the Custom Columns Definition editor and modify the layout's custom columns definition (menu Tools, command Custom Columns Definition...). In the editor go to the Edit menu and select Defines... to open the Defines dialog. By default the eventlog2syslog layout's defines should be configured as shown in Example 14.7. 141/147 USE_BASE_MACROS=1 USE_ALL_MACROS=0 USE_FULLROW_COLUMN=1 USE_UID_COLUMN=1 USE_NEWDATA_COLUMN=0 USE_SYSLOG_COLUMNS=1 USE_NEWREC_COLUMN=1 Example 14.7: Define settings You must also add the custom columns definition files shown in Example 14.8 to the layout's main custom column definition file (“eventlog.cdf” by default) by using the #include directive. The define settings above ensure that the required columns (e.g. the syslog columns) and commands (e.g. the commands which convert some event log columns to syslog columns) are available in the included custom columns definition files shown in Example 14.8. #include #include #include #include <basemacros.cdf> <basecolumns.cdf> <eventlog_basecolumns.cdf> <eventlog_syslog.cdf> Example 14.8: Required custom columns definition files The “eventlog_syslog.cdf” file is listed in Example 14.9. It converts the information of event log columns (e.g. TYPE) to the format expected by syslog (e.g. SYSLOG_PRI). #pragma once #if USE_SYSLOG_COLUMNS == 1 #include <basemacros.cdf> // 0 Emergency: system is unusable // 1 Alert: action must be taken immediately // 2 Critical: critical conditions // Error -> 3 Error: error conditions // Warning -> 4 Warning: warning conditions // 5 Notice: normal but significant condition // Info -> 6 Informational: informational messages // 7 Debug: debug-level messages SYSLOG_PRI = add("128", map(TYPE, "0", "Information", "6", "Warning", "4", "Error", "3")); SYSLOG_TIMESTAMP = fmt("%sT%s.%sZ", xtr(ISO_TIMESTAMP, "", "[^ ]+", ""), xtc(ISO_TIMESTAMP, " ", "[^,]+", ""), xtc(ISO_TIMESTAMP, ",", ".+", "")); SYSLOG_APPNAME = rpl(_SOURCE, "\"", "", "[^a-zA-Z0-9_]", "_"); #endif Example 14.9: "eventlog_syslog.cdf" file Besides the configuration settings you also have to schedule the search, refresh and convert task. In the eventlog2syslog layout they should already be scheduled in the appropriate way. However, if they are not you must schedule them yourself. Open the Tools menu and configure the tasks as 142/147 shown in the Scheduled Tasks dialog in Figure 14.6. The “.” refers to the data folder of Ascolog Insight. How to schedule tasks is described in detail in 9 Task Scheduling. The refresh task is directly configured in the Scheduled Tasks dialog (check boxes at the bottom). The configuration of the search task is a little bit more complex. Select the search task and press the Edit button. This will open the Schedule Find in Files dialog. In this dialog you must specify the path to the “eventlog.ini” file in the Look In field. Figure 14.6: Configuration of the Scheduled Tasks dialog for eventlog2syslog layout Then select the convert task and press Edit to open the Schedule Convert Logfile dialog shown in Figure 14.7. In the File name edit control you must specify the location of the “syslog.ini” file because the syslog server is the destination of the conversion. The Encoding field must be set to SYSLOG. The Record definition defines what columns will be sent to the syslog server. Remember that the conversion of event log information to syslog information is done in the custom columns definition (see Example 14.9). The convert task's configuration is also available as persistent record definition. The definition has the name SYSLOG. 143/147 Figure 14.7: Schedule Convert Logfile dialog The syslog client is now ready to use because the eventlog2syslog layout and the default “eventlog.ini” and “syslog.ini” already provide the right settings for a localhost installation. Example 14.10 shows the default settings for the syslog client (section SYSLOG_SENDER). The client will send syslog messages to the port 514 of the localhost (=IP address 127.0.0.1:514). 144/147 [SYSLOG_SENDER] address = 127.0.0.1:514 send_delay = 0 Example 14.10: "Syslog.ini" file which configures the client/sender If you are running the server and the client on the same computer as in the sample of this chapter you will have only one “syslog.ini” file which configures both the syslog client and the syslog server but in different sections (SYSLOG_SENDER respectively SYSLOG_RECEIVER). If you are running the syslog server and the syslog client on two different computers you have to configure two different “syslog.ini” files, one on each computer. The “eventlog.ini” specifies how to access the local event logs. The default settings are shown in Example 14.11. [EVENTLOG] sourcename = Application ; sourcename = System max_records = 1000 Example 14.11: "Eventlog.ini" file which configures how to access the event logs If the scheduled tasks are correctly configured you must enable them if they are not enabled yet. Open the Tools menu and select Enable Scheduled Tasks. Task scheduling is enabled if the Enable Scheduled Task command is displayed in a light gray font. If everything is correct the path to the “eventlog.ini” file is displayed in the Files window and the status is Loaded. 145/147 15 Ascolog Insight as a Windows® service Purchasing an Ascolog Insight Professional license does not entitle you to use the Ascolog Insight SDK or to run Ascolog Insight as a Windows® service. If you are interested in this functionality please contact us to learn more or to get a quote. 146/147 16 Ascolog Insight's Software Development Kit (SDK) Ascolog Insight offers a SDK so that it is possible to develop plug-ins to enhance Ascolog Insight's functionality and to adapt it to your needs. There are three types of enhancements: 1. Decoders enhance the input functionality (e.g. reading logs from a database) 2. Encoders enhance the output functionality (e.g. writing logs to a database) 3. Commands enhance the functionality of Ascolog Insight (e.g. additional analyze functions) In order to extend Ascolog Insight you have to purchase a license for the software development kit of Ascolog Insight. Purchasing an Ascolog Insight Professional license does not entitle you to use the Ascolog Insight SDK or to run Ascolog Insight as a Windows® service. If you are interested in this functionality please contact us to learn more or to get a quote. 147/147 ">
/
डाउनलोड
बस एक दोस्ताना अनुस्मारक. आप दस्तावेज़ को यहीं देख सकते हैं. लेकिन सबसे महत्वपूर्ण बात, हमारे एआई ने इसे पहले ही पढ़ लिया है. यह जटिल चीजों को सरल शब्दों में समझा सकता है, किसी भी भाषा में आपके सवालों के जवाब दे सकता है और सबसे लंबे या सबसे जटिल दस्तावेजों को भी जल्दी से नेविगेट करने में आपकी मदद कर सकता है.
विज्ञापन