Nimsoft® Monitor™
data_engine Guide
v7.6 series
Legal Notices
Copyright © 2012, Nimsoft. All rights reserved.
Warranty
The material contained in this document is provided "as is," and is subject to being changed, without notice, in future editions.
Further, to the maximum extent permitted by applicable law, Nimsoft Corporation disclaims all warranties, either express or
implied, with regard to this manual and any information contained herein, including but not limited to the implied warranties of
merchantability and fitness for a particular purpose. Nimsoft Corporation shall not be liable for errors or for incidental or
consequential damages in connection with the furnishing, use, or performance of this document or of any information
contained herein. Should Nimsoft Corporation and the user have a separate written agreement with warranty terms covering
the material in this document that conflict with these terms, the warranty terms in the separate agreement shall control.
Technology Licenses
The hardware and/or software described in this document are furnished under a license and may be used or copied only in
accordance with the terms of such license.
No part of this manual may be reproduced in any form or by any means (including electronic storage and retrieval or translation
into a foreign language) without prior agreement and written consent from Nimsoft Corporation as governed by United States
and international copyright laws.
Restricted Rights Legend
If software is for use in the performance of a U.S. Government prime contract or subcontract, Software is delivered and
licensed as "Commercial computer software" as defined in DFAR 252.227-7014 (June 1995), or as a "commercial item" as
defined in FAR 2.101(a) or as "Restricted computer software" as defined in FAR 52.227-19 (June 1987) or any equivalent agency
regulation or contract clause. Use, duplication or disclosure of Software is subject to Nimsoft Corporation’s standard
commercial license terms, and non-DOD Departments and Agencies of the U.S. Government will receive no greater than
Restricted Rights as defined in FAR 52.227-19(c)(1-2) (June 1987). U.S. Government users will receive no greater than Limited
Rights as defined in FAR 52.227-14 (June 1987) or DFAR 252.227-7015 (b)(2) (November 1995), as applicable in any technical
data.
Trademarks
Adobe®, Acrobat®, Acrobat Reader®, and Acrobat Exchange® are registered trademarks of Adobe Systems Incorporated.
Intel® and Pentium® are U.S. registered trademarks of Intel Corporation.
Java(TM) is a U.S. trademark of Sun Microsystems, Inc.
Microsoft® and Windows® are U.S. registered trademarks of Microsoft Corporation.
Netscape(TM) is a U.S. trademark of Netscape Communications Corporation.
Oracle® is a U.S. registered trademark of Oracle Corporation, Redwood City, California.
UNIX® is a registered trademark of the Open Group.
Contact Nimsoft
For your convenience, Nimsoft provides a single site where you can access information
about Nimsoft products.
At http://support.nimsoft.com/, you can access the following:
■
Online and telephone contact information for technical assistance and customer
services
■
Information about user communities and forums
■
Product and documentation downloads
■
Nimsoft Support policies and guidelines
■
Other helpful resources appropriate for your product
Provide Feedback
If you have comments or questions about Nimsoft product documentation, you can
send a message to support@nimsoft.com.
Contents
Chapter 1: data_engine 7.6
7
Introduction ................................................................................................................................................................. 8
Prerequisites ................................................................................................................................................................ 9
data_engine Configuration........................................................................................................................................... 9
The General Tab .................................................................................................................................................. 10
The Database Tab ................................................................................................................................................ 14
The Quality of Service Tab .................................................................................................................................. 16
The Schedule tab ................................................................................................................................................. 17
Advanced Configuration ............................................................................................................................................. 18
Raw Configuration............................................................................................................................................... 18
Basic Benchmarking ................................................................................................................................................... 22
Handling messages from data_engine queue ..................................................................................................... 23
Handling QoS Definitions .................................................................................................................................... 23
D_QOS_PROBES .................................................................................................................................................. 23
QoS Objects ......................................................................................................................................................... 24
Data_engine Starting Up ..................................................................................................................................... 25
Flushing Data to SQL Server Rules ...................................................................................................................... 26
Operating Mode Flags ......................................................................................................................................... 26
The Callback Log-statistics .................................................................................................................................. 27
Configuration Parameters ................................................................................................................................... 31
The Benchmark Tab ............................................................................................................................................ 31
How to Test a Queue File .................................................................................................................................... 32
Contents 5
Chapter 1: data_engine 7.6
This description applies to data_engine probe version 7.6.
This section contains the following topics:
Documentation Changes (see page 8)
Introduction (see page 8)
Prerequisites (see page 9)
data_engine Configuration (see page 9)
Advanced Configuration (see page 18)
Basic Benchmarking (see page 22)
Chapter 1: data_engine 7.6 7
Introduction
Documentation Changes
This table describes the version history for this document.
Version
Date
What's New?
7.6
October 2010
Changed implementation (Oracle) from using direct
path insert to array based inserts, which has
increased performance and stability.
7.5
August 2010
Added basic benchmarking capabilities to
data_engine for nimsoft use. Ported code to
Windows 64.
Related Documentation
Documentation for other versions of the data_engine probe
Getting Started with Nimsoft® Probes
Nimsoft® Probes Reference
Introduction
The Nimsoft Data Engine has two primary focus areas. These areas are:
■
Subscribes to Quality of Service messages
■
Inserts QoS data into the database
In order to reduce the network traffic, the Data Engine should run on a Hub as close to
the database server as possible.
A subscriber channel is opened to the primary Hub. This hub will be referred to as the
SLM Hub. QoS messages are fed to the Data Engine using this channel. The incoming
messages will be reacted upon, and database operations will be performed accordingly.
8 data_engine Guide
Prerequisites
Prerequisites
The appropriate version of the compat-libstdc++ runtime library is needed to be
installed on your system as a prerequisite for the probe. This is required for C++ runtime
compatibility.
The copy of distribution specific compat-libstdc++ runtime library can be obtained from
http://rpm.pbone.net http://rpm.pbone.net/ or you may also obtain a copy of this file
from http://rpmfind.net http://rpmfind.net
For CentOS, go to Programs > System > Add/Remove Software.
■
Select the "Search" tab and search for "compat-libstdc" and use "33" library.
■
For "5" compatibilities, select the appropriate version of the compat-libstdc++
library for your distribution and install it.
data_engine Configuration
You can configure the data_engine by double-clicking the probe in the Infrastructure
Manager. This opens the data_engine dialog.
The configuration user-interface shows the following tabs:
■
General
■
Database
■
Quality of Service
■
Schedule
Chapter 1: data_engine 7.6 9
data_engine Configuration
The General Tab
Field
Description
Data Management
Delete raw data ...
Delete data from raw data tables when data is older than N days.
Delete historic data
Delete data from the historic tables when data is older than N days.
Summarize raw data
Summarize data from raw data after N hours.
Automatically reindex
tables
Turning on this option, the data_engine will reindex the QoS tables in the database.
This will improve performance for all the applications that consumes QoS data,
including the delete expired data routines.
The operation will run at the same time as the other maintenance routines, which
by default are executed at regular intervals each 24 hours, and are by default
started at 00:40
These parameters may be changed by entering the raw configurator for the
data_engine and changing the values for the data_management_time_spec key.
Index maintenance
properties
Log Setup
10 data_engine Guide
Click the Advanced… button to activate or deactivate the index maintenance
properties. For more details, please refer to the Index maintenance properties
section.
data_engine Configuration
Field
Description
Log Level
Sets the level of details written to the log file. Log as little as possible during normal
operation to minimize disk consumption, and increase the amount of detail when
debugging.
Log File
The file where the probe logs information about its internal activity.
Status button
Opens the Status window, reflecting the current work status of the probe. For more
details, please refer to The Status window section.
Statistics
Displays the current transmission rate for transferring QoS messages into the
database
Index maintenance properties
To configure the index maintenance properties, click the Advanced button in the
General tab. The following dialog appears.
Field
Description
Maintenance mode
Controls how to maintain the indexes. The drop-down provides three options:
Dynamic, Reorganize, Rebuild
Dynamic
On choosing this option, the index will be maintained based on the index statistics.
Reorganize
On choosing this option, the maintenance is performed using the "alter index …
reorganize" script of SQL Server.
Rebuild
On choosing this option, the maintenance is performed using "alter index … rebuild"
script of SQL Server.
Chapter 1: data_engine 7.6 11
data_engine Configuration
Field
Description
Online mode
Controls the effect of maintenance on concurrent use of the QoS tables. The
drop-down provides three options: Dynamic, Online, Offline
Please note that Online mode offers greater concurrency but demands more
resources.
This mode is only applicable when Rebuild option is used for maintenance mode
(see above).
Dynamic
On choosing this option, the maintenance is determined by the edition of SQL
Server.
If SQL Server is of Enterprise Edition, then online mode will be used for maintenance
(if chosen maintenance mode supports it). Otherwise, offline will be used.
Online
On choosing this option, the QoS tables are available for update and query during
the table maintenance period.
Offline
On choosing this option, the QoS tables are unavailable for update and query during
the table maintenance period.
Fragmentation level
The fragmentation level information is used if Index name pattern is anything other
than All.
Low threshold
Controls if maintenance will be executed for an index.
If the fragmentation for an index is less than the low threshold value then no
maintenance will be performed.
High threshold
If Dynamic Maintenance mode is chosen and fragmentation is between low and high
threshold, then the Reorganize mode will be used, otherwise the Rebuild mode will
be used.
Index name pattern
This field determines which indexes will be maintained.
Default is blank which results in all indexes being considered for maintenance.
The keyword All can also be specified and causes all indexes for the QoS table to be
maintained using the alter index all on …. . script of SQL Server.
In this case, the settings for Fragmentation level will be ignored since all indexes for
the table are reorganized/rebuilt in a single step.
You can also use any pattern using the SQL Server pattern matching functionality to
filter the indexes to be maintained.
For example, specifying RN%Idx1 would limit maintenance to the index
RN_QOS_DATA_xxxx_Idx1 on the detailed QoS tables.
12 data_engine Guide
data_engine Configuration
The Status window
Clicking the Status button opens the Quality of Service Status window.
The window reflects the current work status of the probe, such as how much data is
stored in the different tables and how many messages it received since the last restart.
The Quality of Service Status window displays the following columns:
■
QoS Name
■
Table Name
■
# Raw Rows
■
KB Raw Data
■
# Historic Rows
■
KB Historic Data
Chapter 1: data_engine 7.6 13
data_engine Configuration
The Database Tab
The Database tab enables you to specify the database connection settings. The
currently supported databases are Microsoft SQL Server and MySQL.
The database settings specified here are used by various other probes, such as
dashboard_engine, discovery_server, sla_engine, wasp, dap, etc.
For MS-SQL:
Field
Description
Database vendor
Select the name of the vendor of the database. For MS-SQL,
select Microsoft option.
Connection
Information
14 data_engine Guide
Provider
The ADO provider used for the database connection
Initial Catalog
The database name
Data Source
The database server
User ID
The login user
Password
The login user’s password. Ensure that the password does NOT
contain any special characters (such as ";").
data_engine Configuration
Field
Description
Parameters
Other parameters to the OLEDB connection
Click the Test Connection button to check the current connection setup.
For MySQL:
Field
Description
Database vendor
Select MySQL option.
Connection
Information
Schema
Enter the database schema name
Server Host
Enter the database server name or IP address
Port
Enter the port number to connect to the database server
Username
The login user name
Password
The login user’s password. Ensure that the password does NOT
contain any special characters (such as ";").
Click the Test Connection button to check the current connection setup.
Chapter 1: data_engine 7.6 15
data_engine Configuration
The Quality of Service Tab
The tab contains a list of all the QoS types defined in the database.
Field
Description
QoS Name
The QoS type name. On the form QOS_xxx
QoS Group
The QoS group is a logical group to which the QoS belongs. Optional.
Description
Description of the QoS type.
Unit (abbrev.)
The unit of the QoS data (The abbreviated form of the QoS data unit).
Has Max Value
Has the data type an absolute max. E.g., disk size, memory usage.
Is Boolean
Is the data type a logical type? Example: host is available, printer is up.
Type
Different data types:
0 = Automatic (The sample value is read at fixed intervals, individually set for each of
the probes).
1 = Asynchronous (the sample value is read only each time the value changes, and
the new value is read.
16 data_engine Guide
data_engine Configuration
The Schedule tab
The Schedule tab is used to schedule the process of data maintenance.
Field
Description
Range
Start at
Select the date from the calendar by clicking the dropdown arrow. This date
and time indicates the time when the data maintenance starts.
No end date/End after/End by
Select an option to end the data maintenance process.
■
Select No end date option to continue the data maintenance process
non-stop.
■
If you select the End After option, enter number of occurrences in the
box.
■
If you select the End by option, then select the date from the calendar
by clicking the dropdown arrow.
Pattern
Minutely/Hourly/
Daily/Weekly/
Select the duration pattern at which the data maintenance process should
run.
Monthly/Yearly
Chapter 1: data_engine 7.6 17
Advanced Configuration
Every/ Every <pattern>
For the selected pattern, specify time more precisely. For example, if you
select Minutely:
■
Select Every to set minutes manually by entering the minute in the box.
■
Select Every minute to run the data maintenance every minute.
Advanced Configuration
This section provides advanced configuration settings for the data_engine probe.
Raw Configuration
You can use the Raw Configure option to update advanced configuration settings.
Monitor Interval Configuration
At regular intervals, the data_engine monitors the database server for free database
and disk space. Error messages may occur if the data_engine is not able to access this
information due to limited user rights.
This can be avoided by turning this monitoring off by entering the raw configurator for
the data_engine and setting the monitor_interval key to 0 (the default value is 300
seconds).
18 data_engine Guide
Advanced Configuration
MySQL Client Buffer Configuration
If the data_engine probe is using MySQL database, the Raw Configure dialog also
enables you to specify MySQL parameters, such as mysql_buffer_increase and
mysql_buffer_size.
Chapter 1: data_engine 7.6 19
Advanced Configuration
Update Statistics for Microsoft SQL Server
The Microsoft ® SQL Server contains a feature called Statistics. For basic information
about this feature, please refer to this post:
http://www.developer.com/db/article.php/3622881/Basics-of-Statistics-in-SQL-Server-2
005.htm
Normally, the statistics is managed automatically by the SQL Server. However, in some
cases, this does not work. For example, when performing bulk inserts using OLE DB
FastLoad API, the statistics for data tables are not automatically updated.
This can lead the query optimizer to choose a non-optimal path when doing queries,
which in turn can lead to poor performance and extra work for SQL Server.
Nimsoft have added an option in data_engine to automatically command to update
statistics. The code is in a stored procedure in the SLM database called
‘spn_de_UpdateStatistics’.
The behavior can be controlled with a few variables that can be tuned via the raw
configure.
Key name
Default value
Type
statistics_age
24
Time in hours. This means when
stored procedure is called, statistics
that are older than this number will
be updated. This value is used by the
stored procedure, not data_engine
itself.
If this number is set to 0 (zero),
statistics will be disabled and not be
run at all by the data_engine.
20 data_engine Guide
statistics_pattern
RN_QOS_DATA%
String pattern to which data tables
will be updated.
statistics_loglevel
0
Numbers 0 to 5 used by stored
procedure when logging to
tbnLogging.
Advanced Configuration
statistics_time_pattern
<not set>
The scheduling string that
determines when to run statistics. If
this key is empty or not set, the
same schedule that is defined for
data management will be used. This
means, the statistics will be run
when data_engine has finished
index maintenance and data
management.
If this value is specified to a different
schedule, the statistics will be
updated independently of when
data management is scheduled.
The string will be used by the
Nimsoft calendar scheduling library,
which is used by various Nimsoft
components. It supports RFC2445.
See short example below.
Chapter 1: data_engine 7.6 21
Basic Benchmarking
Some string examples copied from the library help file.
/****************************************************************************************
** nimCalCreate - Creates a handle to a nimCal structure
*****************************************************************************************
** PARAMETERS:
**
char
*pattern
- RFC2445 ,'weekly' or 'dts'
** char
*start
- startdate: yyyy-mm-dd [hh:mm:ss] || NULL
**
: weekly format 1 or 2
**
**
start = 'yyyy-mm-dd [hh:mm:ss]' will expect the 'pattern' to comply with RFC2445.
**
= NULL results in setting start to 'now'
**
e.g.
**
h = nimCalCreate("DTSTART:19970610T090000|RRULE:FREQ=YEARLY;COUNT=10",NULL);
**
h =
nimCalCreate("DTSTART:19970610T090000|RRULE:FREQ=YEARLY;COUNT=10","2007-07-25");
**
** pattern = 'weekly' handles two 'start' formats:
**
1. 0123,10:00,14:00 [,NOT]
(old NAS
format)
**
2. MO=12:00-14:00,15:30-17:00;TU=08:00-16:00 (new, allow 8-16)
**
**
h = nimCalCreate("weekly","012,10:00,14:00");
**
h = nimCalCreate("weekly","MO=12:00-14:00,15:30-17:00;TU=08:00-16:00");
**
h = nimCalCreate("dts","2007-08-20 08:00,2007-08-27 08:00,2007-09-03
08:00,2007-09-10"
**
** Note: Free the handle using nimCalFree.
****************************************************************************************/
You can alternatively create a schedule in NAS and use the resulting string from there or
use data_engine scheduler to create a string if you do not want to code it by hand.
Basic Benchmarking
Nimsoft has built a basic throughput logging capability into the data_engine probe. If
this option is turned on, data_engine will start logging some values to a plain text file,
indicating what is flowing through the data_engine (number of messages), and how
much time data_engine had to spend at various checkpoints.
22 data_engine Guide
Basic Benchmarking
Handling messages from data_engine queue
Messages that come to data_engine can be of two types:
■
QOS messages (samples gathered)
■
QOS definitions
Handling QoS Definitions
All the Nimsoft probes will send QOS definitions when they start up. These messages
travel into data_engine queue and will be processed. If data_engine already knows
these definitions, they will be discarded.
If data_engine does not know the definition (i.e. this instance of data_engine has not
seen this definition yet), it will be single-inserted into S_QOS_DEFINITION table. The
RN_QOS_DATA_xxxx and HN_QOS_DATA_xxxx tables that will belong to this new QOS
definition will be created in the database.
This means that, upon benchmarking with lots of new QOS definitions, there is an extra
overhead of inserting all these rows into S_QOS_DEFINITION in addition to creating the
physical tables in the database.
D_QOS_PROBES
There is a utility logging table called D_QOS_PROBES, which can be populated by
data_engine. It is just a logging table to register the first time a new QOS definition
comes in from a probe, on a given host. However, data_engine performs a SELECT
operation into this table to see if this combination of (host, qos and probe) is unique in
that table. If the row exists, nothing more is done. If the row does not exist, it is logged
by inserting a row into this table.
However, this can result into a big performance drain if you have a lot of new QOS
definitions coming in all the time. Each one will generate a SELECT into D_QOS_PROBES
if this feature is turned on. All the Nimsoft probes usually only send this each time they
start up. But we have seen custom probes that send them more frequently.
Pre-NMS 5.0, this option was always turned on. In NMS 5.0, it is turned off by default.
You can optionally turn it back on by setting a raw configure value in data_engine.
Chapter 1: data_engine 7.6 23
Basic Benchmarking
QoS Objects
Data_engine will cache the entire S_QOS_DEFINITION and S_QOS_DATA tables in
memory to have fast lookup. This can consume a lot of memory, typically 500-800 MiB
with around 700 million to 1 million rows in S_QOS_DATA with around 10-20
definitions.
When a QOS message comes in to data_engine, it will be processed. If there is no
matching QOS definition in S_QOS_DEFINITION table, the QOS sample will be dropped.
It is not much overhead in this.
Other sanity checks will be performed, such as whether the sample requires max value.
It will then be looked up in S_QOS_DATA from memory to see if we have already this
combination of QOS/SOURCE/TARGET. If it is found, we already have a valid table_id
and which table to insert data to.
The sample will be stored temporarily in memory in what we call RN bucket objects. It
will be a collection of samples that are due to be committed to the database server. The
exact way of storing and committing data to the database servers are specific to each
database platform. But the data flow is generically the same for all supported platforms.
If the object do not exist (QOS/SOURCE/TARGET), a new object will have to be created
in S_QOS_DATA. This is also a small overhead if you start a new test where all the
objects do not exist. The next time they do exist.
24 data_engine Guide
Basic Benchmarking
Data_engine Starting Up
When data_engine starts, it will need to load both S_QOS_DATA and
S_QOS_DEFINITION into memory and set up bulk connections to the databases. This will
take some time.
This section provides a basic overview of data_engine’s main activities in the QOS
thread.
1.
Listen to incoming QOS messages from hub (subscribe to data_engine queue)
2.
Validate QOS messages
3.
Sort into different rn bulk objects temporarily
4.
Iterate bulk objects and bulk commit data to database server, sequentially.
Note: These activities are done in sequence and not in parallel.
Chapter 1: data_engine 7.6 25
Basic Benchmarking
Flushing Data to SQL Server Rules
The data to be flushed to database server are determined on two criteria:
1.
If a bucket reaches a configurable amount of messages, default 5.000 messages, it
will be committed to database server.
2.
If the data in a bucket is more than x seconds old (default 5 seconds), it will be
committed to database server.
Note that these rules are checked when QoS thread is iterating over all RN bulk objects
and not during the time it is subscribing to messages.
This means, you may actually get more than 5.000 messages per commit, even if that is
your number. So, if you have > <your limit>, data_engine will commit that bucket to sql
server.
Operating Mode Flags
Data_engine probe has a couple of operating modes. Manipulating these requires
caution and should only be done on test systems where loss of data is not a problem.
The normal steps taken in data_engine are as follows:
1.
Receive data from hub
2.
Unpack qos messages
3.
Process & Validate
4.
Sort messages into rn bulk objects (or buckets)
5.
Commit data to db server (see rules described in this document)
For performance testing purposes, it is possible to skip one or more of these steps, E.g.
if data_engine should not commit data to db server.
Important! The messages will be thrown away. Do not do this on production systems.
The key in the configuration file that controls this flow is ‘benchmark_mode’ in /setup. It
contains a number which is a bitmask flag. The following values are accepted:
Operation
Without logging
With logging
Subscribe to messages
0
16
Unpack qos messages
Process
Sort
Commit
26 data_engine Guide
Basic Benchmarking
Subscribe to messages
1
17
3
19
7
23
15
31
Unpack qos messages
Process
Sort
Commit
Subscribe to messages
Unpack qos messages
Process
Sort
Commit
Subscribe to messages
Unpack qos messages
Process
Sort
Commit
Subscribe to messages
Unpack qos messages
Process
Sort
Commit
There is another flag which controls if the log file should be truncated or appended to.
The number is 32.
For example: If user wants subscribe to messages, unpack qos messages, process (3) +
logging (16) + reset log(32) = 19 + 32 = 51.
■
Output to logfile : 16
■
Reset logfile: 32
User can use this to test fixed qos queue files, either recorded from a live system or
populated by other tools.
The Callback Log-statistics
The callback has been designed to help control output during runtime.
Chapter 1: data_engine 7.6 27
Basic Benchmarking
Enable Throughput Logging
To enable logging to a text file, run the callback log_statistics with the key log_stats =
yes (string).
Note: This setting does not persist through a cold restart.
Viewing Output
The text file will be updated continuously after each qos thread cycle, meaning after 1
round of dispatching messages, 1 round of iterating rn buckets to commit data to
database.
User may want to use a utility which can tail files, otherwise you need to close and
reopen the file.
Checking Current Status
The parameter "report" can be used to print current summary statistics, similar to that
which will be appended to the log file when the logging is ordered to stop. Pass in
log_stats = yes and report = yes while data_engine is running with statistics on.
Sample output will be appended to the log file, and then the test will resume:
--------------------------------[Summarized]----------------------------------451939 68514432
89311 5020755
73598668
236452
--------------------------------[Averages]------------------------------------Time
:
2010-11-02 14:58:33 -> 2010-11-03 11:25:28 (20
hours, 26 minutes, 55 seconds)
Messages
:
451939 messages
Definitions
:
20
Committed
:
236452 rows
28 data_engine Guide
Messages/Dispatcher
Messages/Total
Commit rows/commit time
Commit rows/total time
:
:
:
:
7
6
47
3
Disp
Callback
Commit
:
:
:
93.05 %
0.12 %
6.83 %
messages/s
messages/s
rows/s
rows/s
Basic Benchmarking
Resetting Counters During Runtime
The callback can be used to reset counters back to 0 while still letting the log output
continue. Counters are reset if you stop and start logging as well. It has been added as a
convenient way of setting the counters to 0.
To do this, pass in log_stats = yes, report = no, reset_counter = yes during a time when
the statistics is already running.
Chapter 1: data_engine 7.6 29
Basic Benchmarking
What Does the Log File Contain?
The name of the log file will be the name of the queue + .txt
For example: If user is draining data_engine main queue, the text file will be
‘data_engine.txt’.
The output of the log might look something like this:
AT: 2010-09-03 11:41:19 queue: data_engine flags: Unpack, Process, Sort,
Commit
time
msg per sec
msg count
time dispatcher
time callback
time commit
time total
rows committed buckets flushed
buckets_flushed_size buckets_flushed_time
2010-09-03 11:41:20
0
0
999
0
0
999
0
0
0
0
2010-09-03 11:41:21
0
0
202
0
0
202
0
0
0
0
2010-09-03 11:41:22
18
20
1000
2
50
1063
20
5
0
5
2010-09-03 11:41:23
0
0
999
0
0
999
0
0
0
0
There is basic header information, which includes the following:
1.
Timestamp
2.
Name of queue being processed
3.
Operation mode flags (documented earlier in this document)
There are 11 columns, separated by tabular character, they are:
30 data_engine Guide
1.
Time: The time this line in the log was written.
2.
Msg per sec: A calculated value showing current throughput (messages per second,
based on the number of messages last "dispatcher time" divided by "total time")
3.
Msg count: Number of messages received during last "dispatcher time"
4.
Time dispatcher: The amount of milliseconds data_engine used receiving and
processing qos messages from hub during last iteration.
5.
Time callback: The amount of milliseconds data_engine used to validate and sort
qos messages into rn bulk objects. Note: this time is included as part of the
"dispatcher time", which covers both receiving and validating/sorting messages.
6.
Time commit: The amount of milliseconds data_engine used last iteration to bulk
commit data to database server.
7.
Time total: The summarized time in milliseconds data_engine used both in
dispatcher and committing data to database server. This also includes the time
data_engine used to check if connection was still up.
8.
Rows committed: Number of rows committed last iteration
Basic Benchmarking
9.
Buckets flushed: This is the number of rn bulk objects that had data in them that
was just flushed to database server
10. Buckets flushed size: The number of buckets (of the total buckets flushed) that was
flushed because they exceeded the number of messages we need before a bulk
insert is conducted.
11. Buckets flushed time: The number of buckets (of the total buckets flushed) that was
flushed because the age of the data in the buckets exceeded the allowed delay
before committing to database server.
To turn off the logging, user can call the log_statistics callback again and pass in
parameter log_stats = no.
Configuration Parameters
bucket_flush_size = 5000
Number of allowed buckets before its flushed
bucket_flush_time = 5
Number of seconds before flushing a bucket
dispatcher_time = 1000
Number of milliseconds to spend in dispatcher loop
hub_bulk_size = 20
Number of messages to receive each time from hub
The Benchmark Tab
The benchmark tab will not be shown by default. You can enable it by setting a
parameter using the raw configure in data engine’s configuration file.
■
Section: <setup >
■
Key: show_benchmark_tab
■
Value: Yes
Save and the next time you open data engine’s GUI the tab will be available.
Chapter 1: data_engine 7.6 31
Basic Benchmarking
How to Test a Queue File
Before testing such files, it is recommended to take a backup of the database, in order
to provide easy access to restoring database server state. Note: be sure not to have
data_engine insert production data if planning to restore the database.
1.
User needs to upload queue file to data_engine’s folder. A sub folder called
"benchmark" should reside for data_engine v7.60 and later. One can for instance,
use Dr. NimBUS to transfer the file.
Example (files copied to benchmark folder):
2.
32 data_engine Guide
Open the data_engine GUI when data_engine is running. This should display the
‘benchmark’ tab.
Basic Benchmarking
3.
Click List Files button.
4.
The dialog changes to something as shown blow, provided there are queue files in
the benchmark folder.
Chapter 1: data_engine 7.6 33
Basic Benchmarking
5.
User selects which modes to run this test with from the checkboxes under "Test
options".
6.
Select a queue files in "Available queue files", right click and choose "Post file to
hub"
Note: For this test to succeed, data_engine will need admin permissions on the hub. It
will try to do the following.
1.
Copy selected queue file into <hub directory>/q folder (where hub stores queue
files)
2.
Create a new queue on the hub using callbacks in the hub
3.
Make data_engine disconnect default data_engine queue and connect to this test
queue instead.
4.
Drain it empty. Data_engine will call callbacks in hub to monitor how many
messages are left in the queue.
5.
Switch back to main data_engine queue when test queue is empty.
6.
Remove test queue from hub using callbacks on hub.
Data_engine probe does not have permissions to do the above operations on the hub.
User will need to go to Probe Administration in Infrastructure Manager and change the
permission for data_engine from "read" to "admin".
User can change it back to "read" when the test is complete.
When the test completes successfully, a log file has been written in data_engine’s folder
with the name of the queue + .txt extension. If the GUI has not timed out yet, it will also
display a window showing the content of the log file.
Example:
The data_engine log might look like this when a test is ordered from the GUI:
34 data_engine Guide
Basic Benchmarking
Sep 6 11:08:34:635 [4820] de: post_to_hub from 193.71.55.115/1705
Sep 6 11:08:34:651 [6204] de: do_post_to_hub - using flags from caller: 0
(None)
Sep 6 11:08:34:651 [6204] de: do_post_to_hub - copying from
benchmark\32bit_test120_q_data_engine.sds to
..\..\..\hub\q\q_32bit_test120_q_data_engine.sds
Sep 6 11:08:35:823 [6204] de: do_post_to_hub - copy complete
Sep 6 11:08:35:823 [6204] de: do_post_to_hub - trying to get session to hub
Sep 6 11:08:35:823 [6204] de: do_post_to_hub - got session to hub
Sep 6 11:08:35:823 [6204] de: do_post_to_hub - creating queue name:
'32bit_test120_q_data_engine'
Sep 6 11:08:36:464 [6204] de: do_post_to_hub - queue created OK
Sep 6 11:08:36:823 [6204] de: do_post_to_hub - sleeping for 6 seconds, giving
hub time to restart...
Sep 6 11:08:42:870 [6204] de: do_post_to_hub - continue...
Sep 6 11:08:42:870 [6204] de: do_post_to_hub - changing queue to
'32bit_test120_q_data_engine'
Sep 6 11:08:42:917 [7536] de: dispatch: callback 'hubpost_bulk' return
non-zero value 7
Sep 6 11:08:43:917 [7536] de: Total - 205669 messages; 0 msg/s
Sep 6 11:08:44:667 [4820] de: get_stats - from 193.71.55.115/1721
Sep 6 11:08:45:886 [7536] de: SubscribeCallback - disconnected from hub
[queue: data_engine]
Sep 6 11:08:58:886 [7536] de: qos_check - subscriber attached to queue: OK
(bulk size=21) (queue: 32bit_test120_q_data_engine)
Sep 6 11:08:59:074 [6204] de: do_post_to_hub - queue has 120001 messages left
Sep 6 11:09:00:120 [6204] de: do_post_to_hub - queue has 119518 messages left
Sep 6 11:09:01:230 [6204] de: do_post_to_hub - queue has 105406 messages left
Sep 6 11:09:02:433 [6204] de: do_post_to_hub - queue has 91105 messages left
Sep 6 11:09:03:574 [6204] de: do_post_to_hub - queue has 76951 messages left
Sep 6 11:09:04:652 [6204] de: do_post_to_hub - queue has 60466 messages left
Sep 6 11:09:05:667 [6204] de: do_post_to_hub - queue has 42406 messages left
Sep 6 11:09:06:699 [6204] de: do_post_to_hub - queue has 22708 messages left
Sep 6 11:09:07:761 [6204] de: do_post_to_hub - queue has 7231 messages left
Sep 6 11:09:08:761 [6204] de: do_post_to_hub - queue has 0 messages left
Sep 6 11:09:08:761 [6204] de: do_post_to_hub - removing queue
'32bit_test120_q_data_engine' from hub
Sep 6 11:09:08:777 [6204] de: do_post_to_hub - queue removed
Sep 6 11:09:08:777 [6204] de: do_post_to_hub - sleeping for 6 seconds
Sep 6 11:09:14:668 [4820] de: get_stats - from 193.71.55.115/1762
Sep 6 11:09:14:824 [6204] de: do_post_to_hub - point back to normal
data_engine queue
Sep 6 11:09:14:855 [7536] de: SubscribeCallback - disconnected from hub
[queue: 32bit_test120_q_data_engine]
Sep 6 11:09:16:027 [6204] de: do_post_to_hub - finished (OK:0)
Sep 6 11:09:16:043 [6204] de: log Size=6392
Sep 6 11:09:16:043 [6204] de: log
PDS_PCH
6381 AT: 2010-09-03
11:39:18 queue:~
Sep 6 11:09:16:152 [7536] de: qos_check - subscriber attached to queue: OK
(bulk size=21) (queue: data_engine)
Sep 6 11:09:16:152 [7536] de: Total - 325670 messages; 3750 msg/s
Sep 6 11:09:19:558 [7536] de: Commit - inserted
1 rows to RN_QOS_DATA_0015
in
7 ms (ms/r:7)
Chapter 1: data_engine 7.6 35
Sep 6 11:09:19:574 [7536] de: Commit - inserted
1 rows to RN_QOS_DATA_0012
in
7 ms (ms/r:7)
Sep 6 11:09:19:574 [7536] de: Commit - inserted
2 rows to RN_QOS_DATA_0013
in
6 ms (ms/r:3)
Sep 6 11:09:19:590 [7536] de: Commit - inserted
1 rows to RN_QOS_DATA_0014
Basic Benchmarking
The window in data_engine GUI might contain information like this:
The information from such a test can be copied to a spreadsheet. User can then analyze
the numbers provided for charts or other calculations.
GUI will time out if the test takes more than 5 minutes to complete. There is no option
to change this timeout value.
You cannot abort a test. But you can make it end early by bringing up the hub GUI,
locate your test queue and right click and choose "Empty" to empty the queue.
36 data_engine Guide
Download PDF
Similar pages