Neolane v6.1 - Production Guide

Neolane v6.1 - Production Guide
Production Guide
Neolane v6.1
This document, and the software it describes, are provided subject to a License Agreement and may not be used or copied outside of the
provisions of the License Agreement. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any
form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of Neolane.
The information contained in this document is provided for informational purposes only and may be revised without notice. It does not
constitute a commitment on the part of Neolane. Neolane does not guarantee the accuracy nor the completeness of the information
contained within this document. References to company names are intended to be ficticious and for illustrative purposes only and do not
refer to any real-world company.
Any brands cited are the property of their respective owners. Windows is the registered trademark of Microsoft Corporation in the United
States and other countries. Java, MySQL and Open Office are trademarks of Oracle Corporation in the United States and in other countries.
Linux is the registered trademark of Linus Torvalds in the United States and in other countries. This product includes software developed
by Apache Software Foundation (http://www.apache.org/).
For any questions or queries, please send a message to the following address: doc@neolane.com.
Version number : 8142
Neolane
18 rue Roger Simon Barboux, 94110 Arcueil - France
+33 1 41 98 35 35
www.neolane.com
Table of Contents
Neolane v6.1 - Production Guide
Chapter 1. General architecture of Neolane . . . . . . . . . . . . . . .
5
Minimum architecture . . . . . . . . . . . . . . . . . . . . . . . .
Distributed architecture . . . . . . . . . . . . . . . . . . . . . . . .
List of open ports . . . . . . . . . . . . . . . . . . . . . . . . . .
5
6
6
Chapter 2. Production procedures . . . . . . . . . . . . . . . . . .
9
Configuration principle . . . . . . . . . . . . . . . . . . . . . . . .
Operating principle . . . . . . . . . . . . . . . . . . . . . . . . .
Administration . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . .
Changing the syslogd listening port . . . . . . . . . . . . . . . . .
Configuring security zones . . . . . . . . . . . . . . . . . . . . .
Log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Monitoring Neolane processes . . . . . . . . . . . . . . . . . . . .
Manual monitoring . . . . . . . . . . . . . . . . . . . . . . . .
SMTP Reports . . . . . . . . . . . . . . . . . . . . . . . . .
Automatic monitoring . . . . . . . . . . . . . . . . . . . . . . .
Automatic monitoring via Neolane scripts . . . . . . . . . . . . . . .
Usual commands . . . . . . . . . . . . . . . . . . . . . . . . .
Monitoring commands . . . . . . . . . . . . . . . . . . . . . .
Module launch commands . . . . . . . . . . . . . . . . . . . . .
Shut down services . . . . . . . . . . . . . . . . . . . . . . .
Restart services . . . . . . . . . . . . . . . . . . . . . . . . .
The config command . . . . . . . . . . . . . . . . . . . . . . .
9
10
12
13
13
13
14
15
15
20
24
27
30
30
31
31
31
32
Chapter 3. Data processing . . . . . . . . . . . . . . . . . . . . .
33
Backup . . . . . . . . . . . . . . .
Physical files . . . . . . . . . . . .
Database . . . . . . . . . . . . .
Restoration . . . . . . . . . . . . . .
Duplicating environments . . . . . . . .
Introduction . . . . . . . . . . . .
Implementation . . . . . . . . . . .
Description of the database Cleanup workflow
Introduction . . . . . . . . . . . .
33
33
34
34
34
34
35
38
38
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Neolane v6.1 - Production Guide | 3
Neolane
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Tasks carried out by the Database cleanup workflow . . . . . . . . . . . . . .
38
40
Chapter 4. Updating Neolane . . . . . . . . . . . . . . . . . . . . . . .
49
Upgrade . . . . . . . . . . . . . . . . . .
In Windows . . . . . . . . . . . . . . . .
In Linux . . . . . . . . . . . . . . . . .
Resolving upgrade conflicts . . . . . . . . . .
Informing client consoles of the available update . .
Example of the Unicode switch of an existing instance
.
.
.
.
.
.
49
50
51
52
53
54
Chapter 5. Database maintenance . . . . . . . . . . . . . . . . . . . . .
57
List of tables to maintain . . . . .
Neolane tables . . . . . . .
Customer tables . . . . . . .
Types of maintenance . . . . . .
Application maintenance . . . .
Technical maintenance . . . .
RDBMS Specific recommendations .
PostgreSQL . . . . . . . . .
Oracle . . . . . . . . . . .
DB2 . . . . . . . . . . . .
SQL Server . . . . . . . . .
MySQL . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
57
58
60
61
61
61
62
62
67
68
68
76
Chapter 6. Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . .
79
Temporary files . . . . . . . . .
Database performances . . . . . .
Configuration . . . . . . . . .
Platform configuration . . . . . .
Database maintenance . . . . .
Specifics . . . . . . . . . . .
Modules and frequent issues . . . .
Log precision . . . . . . . . . .
Forwarding of tracking logs . . . . .
OpenOffice . . . . . . . . . . .
Email delivery . . . . . . . . . .
Workflow execution . . . . . . . .
Failure to connect . . . . . . . .
Connection thresholds . . . . . . .
Dr Watson in Windows XP (client) . .
Stack trace in Linux . . . . . . . .
Encoding of the Oracle database . . .
Reactivating the console update request
Lost password . . . . . . . . . .
Abnormal JSP behavior . . . . . .
79
80
80
80
80
81
81
82
83
84
84
84
85
86
86
87
88
88
89
89
4 | © Neolane 2013
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
CHAPTER 1
General architecture of
Neolane
Table of Contents
Minimum architecture . . . . . . . . . . . . . . . . . . . . . . . . .
Distributed architecture . . . . . . . . . . . . . . . . . . . . . . . .
List of open ports . . . . . . . . . . . . . . . . . . . . . . . . . .
5
6
6
This chapter details the general architecture of Neolane.
Minimum architecture
In a minimum configuration, Neolane operates with:
n
n
the Neolane application server,
the database.
Neolane v6.1 - Production Guide - General architecture of Neolane | 5
Neolane
This diagram shows that the only traffic involved in the context of a minimum architecture is:
1 HTTP protocol traffic to the Neolane server via the Internet,
2 SMTP protocol traffic from and to the Neolane server via the Internet.
Distributed architecture
Neolane is made up of multiple modules which can be broken down over several machines. This operating
mode has several advantages:
n
n
n
load balancing,
setting up of module redundancy,
building of an architecture broken down over several service providers (segmentation of the services
provided).
The distribution of modules over several machines provides great flexibility of use and improved adaptability.
Tip:
For more on the various architectures, refer to the Installation Guide.
List of open ports
Port number
Concerned Neolane module or ap- Configurable
plication
443/tcp or 80/tcp
Web Servers (Apache/IIS)
YES
6025/tcp
OpenOffice.org server (Xvfb)
NO
6666/udp (local)
Neolane: Syslogd
YES
8005/tcp (local)
Neolane: web module
YES
6 | © Neolane 2013
General architecture of Neolane
Port number
Concerned Neolane module or ap- Configurable
plication
8080/tcp
Neolane: web module (tomcat)
YES
7777
Statistics server (stat server)
YES
Neolane v6.1 - Production Guide - General architecture of Neolane | 7
Neolane
8 | Neolane v6.1 - Production Guide
CHAPTER 2
Production procedures
Table of Contents
Configuration principle . . . . . . . . . . . . . . . . . . . . . . . . .
Operating principle . . . . . . . . . . . . . . . . . . . . . . . . .
Administration . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . .
Changing the syslogd listening port . . . . . . . . . . . . . . . . . . .
Configuring security zones . . . . . . . . . . . . . . . . . . . . . .
Log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Monitoring Neolane processes . . . . . . . . . . . . . . . . . . . . . .
Manual monitoring . . . . . . . . . . . . . . . . . . . . . . . . .
SMTP Reports . . . . . . . . . . . . . . . . . . . . . . . . . .
Automatic monitoring . . . . . . . . . . . . . . . . . . . . . . . .
Automatic monitoring via Neolane scripts . . . . . . . . . . . . . . . . .
Usual commands . . . . . . . . . . . . . . . . . . . . . . . . . .
Monitoring commands . . . . . . . . . . . . . . . . . . . . . . .
Module launch commands . . . . . . . . . . . . . . . . . . . . . .
Shut down services . . . . . . . . . . . . . . . . . . . . . . . .
Restart services . . . . . . . . . . . . . . . . . . . . . . . . . .
The config command . . . . . . . . . . . . . . . . . . . . . . . .
9
10
12
13
13
13
14
15
15
20
24
27
30
30
31
31
31
32
Configuration principle
The Neolane platform is based on the concept of instances, similar to that of virtual hosts used by
Apache. This mode of operation lets you share a server by assigning several instances to it. Instances
are completely separate from each other and operate with their own database and configuration
file.
For a given server, there are two elements that are common to all Neolane instances:
n
The internal password: this is the general administrator password. It is common to all instances
of a particular application server.
Warning:
To log on with the Internal identifier, you need to have defined a password beforehand. For
more on this, refer to the Installation guide.
n
Multiple technical server configurations: these configurations can all be overloaded in the specific
configuration of an instance.
Neolane v6.1 - Production Guide - Production procedures | 9
Neolane
The configuration files are saved in the conf directory of the installation directory. The configuration is broken
down into three files:
n
n
n
serverConf.xml: overall configuration for all instances.
serverConf.xml.diff: delta between the initial configuration and the current configuration. This file is
generated automatically by the application and must not be modified manually. It is used to automatically
propagate user modifications when updating a build version.
config-<instance>.xml (where <instance> is the instance name): specific configuration of an
instance.
An instance configuration is loaded as follows:
1 The module loads the serverConf.xml file to obtain the parameters shared by all instances.
2 It then loads the config-<instance>.xml file. The values found in this file have priority over values
contained in serverConf.xml.
These two files have the same format. Any value in serverConf.xml can be overloaded for a given
instance in the config-<instance>.xml file.
This operating mode provides great flexibility for configurations.
Operating principle
Technically, the Neolane platform is based on several modules.
There are many Neolane modules. Some operate continuously, while others are started up occasionally to
perform administrative tasks (e.g. to configure the database connection) or to run a recurrent task (e.g.
consolidating tracking information).
There are three types of Neolane modules:
n
n
n
Multi-instance modules: a single process is run for all instances. This applies to the following modules:
web, syslogd, trackinglogd and watchdog (activities from the config-default.xml file).
Mono-instance modules: one process is run per instance. This applies to the following modules: mta,
wfserver, inMail, sms and stat (activities from the config-<instance>.xml file).
Utility modules: these are modules that are run occasionally to perform occasional or recurrent operations
(cleanup, config, downloading tracking logs, etc.).
Module administration is performed using the command line tool nlserver installed in the bin directory of
the installation folder.
The general syntax of the nlserver tool is as follows:
nlserver <command> <command arguments>
For the list of available modules, use the nlserver command.
The available modules are detailed in the following table:
Command
Description
aliasCleansing
Standardizing enumeration values
billing
Sending the system activity report to <billing@neolane.com>
cleanup
Cleansing the database: deletes obsolete data from the database and runs an update of the statistics used
by the database engine optimizer.
config
Modifying server configuration
copybase
Copy of a database
export
Exporting to command line: lets you send to the command line an export model created in the Neolane
client console
fileconvert
Converting a set size file
import
Importing to command line: lets you send to the command line an import model created in the Neolane
client console.
inMail
Inbound mail analyzer
installsetup
Availability of the customer installation file
javascript
Executing JavaScript scripts, with access to SOAP APIs.
10 | © Neolane 2013
Production procedures
Command
Description
job
Command line processing
merge
Form merge
midSourcing
Recovery of delivery information in mid-sourcing mode
monitor
XML Displaying of the status of server processes and scheduled tasks, by instance.
mta
Main Agent transfer message
package
Importing or exporting entity package files
pdump
Displaying server process statuses
prepareda
Preparing a delivery action
restart
Partial server restart
runwf
Execution of a workflow instance
shutdown
Full system shutdown
sms
SMS notification processing
sql
SQL script execution
start
Additional starts
stat
Maintains MTA connection statistics
stop
Partial system shutdown
submitda
Submitting a delivery action
syslogd
Log and trace writing server
tracking
Consolidating and retrieving tracking logs
trackinglogd
Tracking log writing and purging server
watchdog
Startup and monitoring instance
web
Application server (HTTP and SOAP)
wfserver
Workflow server
Warning:
There is one last module: the tracking and relay module linked to the application server which, for the sake
of performance, is integrated via native mechanisms into an Apache or IIS web server via a dynamic library.
There is no Neolane command enabling you to start or administer this module. You must therefore use the
commands of the Web server itself.
Module usage and the syntax of its parameters are displayed using the following command: nlserver
[module] -?
Example:
nlserver config -?
Usage: nlserver [-verbose:<verbose mode>] [-?|h|H] [-version] [-noconsole]
[-tracefile:<file>] [-tracefilter:<[type|!type],...>]
[-instance:<instance>] [-low] [-high] [-queryplans] [-detach]
[-internalpassword:<[password/newpassword]>] [-postupgrade]
[-nogenschema] [-force] [-allinstances]
[-addinstance:<instance/DNS masks[/language]>]
[-setdblogin:<[dbms:]account[:database][/password]@server>]
[-monoinstance]
[-addtrackinginstance:<instance/masks DNS[/databaseId/[/language[/password]]]>]
[-trackingpassword:<[password][/newpassword]>]
[-setproxy:<protocol/server:port[/login]>] [-reload]
[-applyxsl:<schema/file.xsl>] [-filter:<file>]
[-setactivationkey:<activation key>]
[-getactivationkey:<client identifier>]
-verbose : verbose mode
-? : display this help message
-version : display version number
-noconsole : no longer display logs and traces on the console
-tracefile : name of trace file to be generated (without extension)
-tracefilter : filter for the traces to be generated e.g.: wdbc,soap,!xtkquery.
Neolane v6.1 - Production Guide - Production procedures | 11
Neolane
-instance : instance to be used (default instance if this option is not present).
-low : start up with low priority
-high : start up with high priority (not recommended)
-queryplans : generate traces with the execution plans of SQL queries.
-detach : detaches the process from its parent (internal option)
-internalpassword : changes the password of the server internal account.
-postupgrade : updates the database following upgrade to a higher version.
-nogenschema : does not recompute the schemas during database update
-force : updates the database even if this has already been done with the current build
-allinstances : updates the database over all configured instances
-addinstance : adds a new instance.
-setdblogin : sets the parameters for connection to the database of an instance. The DBMS
can be 'oracle', 'postgresql', 'mssql' or 'odbc' (default=postgresql)
-monoinstance : initialises for a single instance ().
-addtrackinginstance : adds a new tracking instance.
-trackingpassword : changes the tracking password of an instance
-setproxy : sets the parameters for connection to a proxy server. The protocol can be
'http', 'https' or 'all'.
-reload : asks the server to reload the configuration of the instances.
-applyxsl : applies an XSL stylesheet to all entities of a schema.
-filter : applies the XTK filter contained in the file during loading of the schema entities.
-setactivationkey : sets the activation key
Administration
Automatic startup of the Neolane modules (web, mta, wfserver, etc.) is provided by the nlserver server.
Installing Neolane automatically configures the machine so that the nlserver service starts up during the
boot sequence.
The following commands are used to start up and shut down the Neolane service manually:
n
In Windows:
n
n
n
net start nlserver6
net stop nlserver6
In Linux (as root):
n
n
/etc/init.d/nlserver6 start
/etc/init.d/nlserver6 stop
Here is a list of the usual administration commands accessible in Linux (as Neolane):
n
Display all started Neolane modules: /etc/init.d/nlserver6 pdump or /etc/init.d/nlserver6
status
Note:
Adding the -who parameter to the pdump command lets you collect information on current connections
(users and processes).
n
Start/stop a multi-instance or mono-instance module (web, trackinglogd, syslogd, mta, wfserver,
inmail):
nlserver start <module>[@<instance>]
nlserver stop <module>[@<instance>][-immediate] [-noconsole]
You can also use the nlserver restart <module>[@<instance>] command to restart a module.
Example:
nlserver start web
nlserver start mta@my_instance
nlserver stop syslogd
nlserver stop wfserver@my_instance
nlserver stop web -immediate
nlserver restart web
12 | © Neolane 2013
Production procedures
Note:
n
n
n
n
If the instance is not specified, the "default" instance will be used.
In the event of an emergency, use the -immediate option to force an immediate halt to the process
(equivalent to Unix command kill -9).
Use the -noconsole option to ensure that the module launched will display nothing on the console.
Its logs will be written to the disk via the syslogd module.
Use the -verbose option to display additional information on process actions.
Example:
nlserver restart web -verbose
nlserver start mta@myinstance -verbose
This option adds additional logs. We recommend starting the processes again without the -verbose
option once you have found the desired information, to avoid overloading logs.
Start up all Neolane processes (equivalent to starting up the nlserver6 service): nlserver watchdog
-noconsole
Shut down all Neolane processes (equivalent to shutting down the nlserver6 service): nlserver
shutdown
Reload the nlserver web module configuration (and the web server extension module, where applicable)
when the serverConf.xml and config-<instance>.xml files have been edited.
n
n
n
nlserver config -reload
Note:
Some configuration changes are not taken into account dynamically; Neolane must be shut down and
then restarted.
Configuration
Changing the syslogd listening port
By default, the syslogd listening port is 666 (udp). You can alter it using an environment variable if necessary.
Once it is configured, this variable is taken into account by all Neolane modules.
In Linux
Edit the customer.sh file and add the following line:
export TRACE_ADDR=localhost:<listening port>
In Windows
You need to create the TRACE_ADDR environment variable with the localhost value: <listening port>.
Warning:
We recommend running some tests to make sure your platform is working after you create this environment
variable.
Configuring security zones
Each operator needs to be linked to a zone to log on to an instance and the operator IP must be included in
the addresses or address sets defined in the security zone. Technical zone configuration is carried out in the
configuration file of the Neolane server. The linking of an operator to a security zone has to be defined in
the console (Administration > Access management > Operators node).
Neolane v6.1 - Production Guide - Production procedures | 13
Neolane
Note:
For more on configuring security zones, refer to the Installation guide.
Log files
The log files are organized as follows:
Each nlserver module generates a log file saved in the following directory: <installation
directory>/var/<instance>/log/<module>.log.
The nlserver syslogd module saves the logs to the disk. This module is similar to the Unix syslog daemon,
but has been adapted for compatibility between Unix and Windows. The other Neolane modules do not save
their logs to the disk; they delegate this task to the syslogd module by sending it UDP packets.
By default, the Neolane platform has the syslogd module installed on it, but it is possible to use another
syslog daemon. This module creates the log files in the log directory.
The logs of multi-instance modules are stored in the following directory: <installation
directory>/var/default/log/. The same log file is shared by all instances (e.g. web.log).
The logs of the other modules are stored in a subfolder named after the instance. Each instance has its own
log files.
Multi-instance log files are listed in the following table:
File
Description
web.log
Web module logs (client console, reports, SOAP API, etc.)
webmdl.log
Logs from the redirection module
watchdog.log
Logs from the Neolane process monitoring module
trackinglogd.log
Tracking logs
The mono-instance log files are listed in the following table:
File
Description
mta.log
mta module logs
mtachild.log
Message delivery processing logs
wfserver.log
Logs of the workflow server module
runwf.log
Workflow execution logs
inMail.log
Bounce mail module log
Warning:
The redir directory only exists on redirection servers. The url subdirectory contains the matches of the
URLs to be redirected, and the subdirectory log contains the tracking logs. To generate tracking logs, the
trackinglogd module must be running.
By default, the logs are limited to two 10 MB files per module and per instance. The second file is called:
<ModuleName>_2.log. The size of the logs is therefore limited to 2*10MB per module and per instance.
14 | © Neolane 2013
Production procedures
You can, however, keep larger files. To enable this, change the value of the maxFileSizeMb="10" setting
in the syslogd node of the conf/serverConf.xml file. This value represents the maximum size in MB of a
log file.
If you wish to maintain further levels of detail in the logs, you can start the Neolane modules with the
-verbose parameter:
nlserver start <MODULE>@<INSTANCE> -verbose
Monitoring Neolane processes
The application server and the redirection server (tracking) can be monitored manually or automatically.
Manual monitoring
Go to the Monitoring universe and click the Overview link to display the Neolane process monitoring page.
The page displayed lets you view the status of the connected instance, i.e.:
n
n
n
information on the instance (version, name, database engine, installed packages, server system indicators),
the list of missing processes and execution information (start date, PID, etc.),
a view of workflows and deliveries.
Neolane v6.1 - Production Guide - Production procedures | 15
Neolane
Log journal
It is possible to display the log journal related to a process. To do this, click on the process, mta for example,
then click Open the log journal.
System indicators
The list of system indicators enables you to display information concerning the machine, such as its physical
and virtual memory, active processes and available disk space. Indicators are different for Linux and Windows
operating systems. Go to the Instance Monitoring page and click the Display link to open the list of
indicators
16 | © Neolane 2013
Production procedures
In Windows
n
n
Pending events queued : indicator specific to Neolane Message Center. Refer to the Events
monitoring thresholds section for more information.
Memory: information concerning the physical memory (RAM).
Current value: actual memory consumption.
Max Value: total amount of memory installed.
Available: amount of available memory.
Warning: this indicator is displayed when memory consumption reaches 80% of the total amount.
Alert: this indicator is displayed when memory consumption reaches 90% of the total amount.
n
When the Warning and Alert indicators are displayed, you can solve the issue by adding RAM to the
machine which the Neolane server is installed on. You can also decide to install the Neolane server on a
dedicated machine.
Swap Memory: information related to the virtual memory that matches a paging file: an area on the
hard disk that Windows uses as if it were RAM.
Current value: actual memory consumption.
Max Value: total amount of memory.
Available: amount of available memory.
Warning: this indicator is displayed when memory consumption reaches 80% of the total amount.
Alert: this indicator is displayed when memory consumption reaches 90% of the total amount.
n
When the Warning and Alert indicators are displayed, you can solve the issue by increasing the size
of the exchange file in the advanced Windows settings.
Disk XXX: information concerning machine readers.
Current value: disk space actually used.
Max Value: total disk capacity.
Available : disk space available
Neolane v6.1 - Production Guide - Production procedures | 17
Neolane
Used: percentage of disk used.
Warning: this indicator is displayed when the available disk space reaches 80% of the total capacity.
n
Alert: this indicator is displayed when the available disk space reaches 90% of the total capacity.
Number of processes too old: information concerning Neolane processes that have been active for
more than one day.
Current value: number of processes currently active.
Max Value: maximum number of authorized processes (1).
Alert: this indicator is displayed if the number of processes equals 1.
When the Alert indicator is displayed, it may be that the concerned process is locked by the SQL database
engine or that it is stuck in an infinite loop. The watchdog process provided by Neolane automatically
re-starts all processes every day and enables you to solve this issue. However, you can also stop the
concerned process yourself to force re-start.
In Linux
n
n
Pending events queued : indicator specific to Neolane Message Center. Refer to the Events
monitoring thresholds section for more information.
Load average (1/5/15 minutes): information concerning the load, i.e. the use rate of the processer
by the processes running on the machine over the last minute, five minutes, or fifteen minutes
Current value: actual load of the machine.
18 | © Neolane 2013
Production procedures
Max value: maximum use load of the process(es) on the machine
Warning: this indicator is displayed when the load reaches 80% of the maximum authorized value over
the last minute, five minutes or fifteen minutes.
n
Alert: this indicator is displayed when the load reaches 90% of the maximum authorized value of the
last minute, five minutes, or fifteen minutes.
Memory: information concerning the physical memory (RAM).
Current value: actual memory consumption.
Max Value: total amount of memory installed.
Available: amount of available memory.
Warning: this indicator is displayed when memory consumption reaches 80% of the total amount.
Alert: this indicator is displayed when memory consumption reaches 90% of the total amount.
n
When the Warning and Alert indicators are displayed, you can solve the issue by adding RAM to the
machine which the Neolane server is installed on. You can also decide to install the Neolane server on a
dedicated machine.
Swap Memory: information related to the virtual memory that matches a paging file: an area on the
hard disk that Windows uses as if it were RAM.
Current value: actual memory consumption.
Max Value: total amount of memory.
Available: amount of available memory.
Warning: this indicator is displayed when memory consumption reaches 80% of the total amount.
Alert: this indicator is displayed when memory consumption reaches 90% of the total amount.
n
When the Warning and Alert indicators are displayed, you can solve the issue by increasing the size
of the exchange file.
Core Files: information concerning the files generated following the crash of a Neolane process. These
files enable you to diagnose the reasons of the crash.
Current Value: number of existing files.
Max Value: maximum number of authorized files (1).
Warning: this indicator is displayed when the number of files nears 1.
Alert: this indicator is displayed when the number of files equals 1.
n
When a process is missing due to a crash, it is shown in red on the list of processes and is re-started
automatically by the watchdog process provided by Neolane.
Number of shared memory segments: information concerning the memory segments shared by all
Neolane processes.
Current value: number of memory segments currently in use.
Max Value: maximum number of memory segments authorized (2).
Warning: this indicator is displayed when the number of memory segments reaches 1.
n
Alert: this indicator is displayed when the number of memory segments reaches 2.
Number of processes too old: information concerning processes that have been active for over one
day.
Current value: number of processes currently active.
Max Value: maximum number of authorized processes.
Warning: this indicator is displayed when the number of processes reaches 80% of the authorized
threshold.
n
Alert: this indicator is displayed when the number of processes reaches 90% of the authorized threshold.
File Handles: information concerning the file descriptors, i.e. the number of files opened per process.
Current value: current number of file descriptors.
Max Value: maximum number of file descriptors authorized by the operating system.
Warning: this indicator is displayed when the number of authorized file descriptors reaches the 80%
threshold.
n
Alert: this indicator is displayed when the number of authorized file descriptors reaches the 90% threshold.
Processes: information concerning the machine processes.
Current value: number of processes currently active.
Max Value: maximum number of authorized processes.
Active Processes: number of active processes.
Neolane v6.1 - Production Guide - Production procedures | 19
Neolane
Inactive Processes: number of inactive processes.
Warning: this indicator is displayed when the number of authorized processes reaches the 80% threshold.
n
Alert: this indicator is displayed when the number of authorized processes reaches the 90% threshold.
Zombie Processes: information concerning the processes that have been stopped but still have a
process identifier (PID) and remain visible in the process table.
Current value: number of zombie processes that are currently active.
Max Value: maximum number of authorize zombie processes (2).
Warning: this indicator is displayed when the number of zombie processes nears 2.
Alert this indicator is displayed when the number of zombie processes reaches 2.
SMTP Reports
SMTP delivery monitoring reports are integrated into the Neolane platform. They can be accessed via the
console or using Web access.
These reports display SMTP delivery statistics and SMTP errors by domain.
To access them, the operator must have Administration rights.
They are grouped in the Monitoring universe, under the 'SMTP Monitoring' label.
Warning:
n
n
Information related to SMTP Monitoring is only available if the email channel has been activated in the
deployment wizard.
The SMTP sending statistics are only offered if the statistics server is launched on the instance.
20 | © Neolane 2013
Production procedures
SMTP sending statistics
The SMTP sending statistics report lets you control server activity. It displays a synthesis of each of the
mtachilds.
The list of indicators for this report is shown below the chart.
1 Total number of messages sent.
2 - Blue line: messages ready for sending which arrived in the Shaper, i.e. last stage before sending SMTP
(coincides with the incoming data).
- Green line: messages successfully sent (coincides with the outgoing data).
- Red line: messages abandoned by the Shaper, returned to the mta (coincides with the data rejected
on this recovery).
3
These values are expressed in number of messages per hour.
Represents two queues of the Shaper:
- Blue curve: queue of active messages. These messages will be sent as soon as possible.
4
- Kaki curve: the 'deferred' queue. These messages cannot be returned for the moment due to throttling
or because no connection to the target is available. Retries will take place every 5s, 10s, 20s, 40s, 2 min,
etc. for the defined MaxAgeSec time before being abandoned.
This charts shows a detail of abandoned messages (red curve on the 2nd chart): it shows the proportion
of messages abandoned without retries (mauve) compared with messages whose sending failed (red).
This lets you view the proportion of messages not processed within the granted period due to limitations
by the statistics server (throttling) or due to remote server unavailability.
SMTP connections open or being opened.
5
6 Estimate of the number of mtachild.
Note:
This report is related to the status of the Email Traffic Shaper component.
SMTP errors per domain
This report lets you view the delivery errors, over a set period, broken down by domain.
Neolane v6.1 - Production Guide - Production procedures | 21
Neolane
Note:
The minConnectionsToLog, minErrorsToLog and minMessagesToLog options of the serverConf.xml
file define the thresholds above which connection statistics are taken into account.
The list of indicators for this report is shown below the table.
n
n
n
n
n
n
n
The Domain column contains the name of the domain to which the messages are sent (or the real
domain name, yahoo.com for yahoo.fr for example),
The Cnx column displays the number of SMTP connections open for this domain,
The Sent column corresponds to the number of messages sent to this domain,
The Volume column displays the volume of messages that have been attempted to be sent to this
domain (approximate value),
The Errors column displays a volume indicator of errors on this domain over the period,
The Last response column displays the last SMTP response message received for this domain,
The Date column displays the date of the last SMTP response received for this domain.
Note:
The values displayed in the Cnx, Sent, and Volume columns are calculated with respect to the period
selected in the Period field.
Click on a domain name to view its errors.
22 | © Neolane 2013
Production procedures
They are categorized by PublicId: this identifier corresponds to an IP address shared by several Neolane
mtas behind a router. The statistics server uses this identifier to memorize the connection and delivery
statistics between this starting point and the target server.
The Owner of domain field lets you group various domain names under the same label. In the initial report
view, all MX domain names will be associated to this owner.
Click on a PublicId identifier to view further detail.
Neolane v6.1 - Production Guide - Production procedures | 23
Neolane
Note:
The percentage of errors is represented by two charts. The first is a horizontal progress bar on a black
background. The second chart is chronological. The selected period is divided into twelve time intervals, each
represented by a vertical progress bar. In both representations, if no error has been detected, the bar is
black. The color of the bar depends on the percentage of errors encountered (yellow, then orange, and lastly,
red). The color grey means that no significant data volume has been found. It is possible to display the exact
percentage of errors by putting the cursor on the chart.
Note:
For further information about SMTP errors and managing them in Neolane, please consult the Installation
guide.
Automatic monitoring
Neolane offers several automatic monitoring methods, which are presented below.
Command line
Command
nlserver monitor
Lets you list a set of indicators on the Neolane modules and the system.
It generates output in an easily processed XML format.
This command can also be run with the -missing parameter, which lists the processes that are missing from
this instance when the configuration files say that they should be executing.
nlserver monitor -missing
10:17:59 >
Application server for Neolane Version 6.0.0 (build 6749) of 26/07/2011
mta@prod
stat@prod
wfserver@prod
Information published by the server
/r/test
The http(s)://<Application server URL/r/test page is used to test the redirection server. We
recommend using this same method to test the frontal servers used for tracking. This page can also be used
to test a load dispatcher.
It displays a line like this in XML format:
<redir status='OK' date='2011-07-27 08:18:11.112Z' build='6743' host='training.neolane.net'
localHost='servername'/>
Frequency: this test does not use any load, and so it can be run very often (e.g. once every second).
/nl/jsp/ping.jsp
This http(s)://<Application server url/nl/jsp/ping.jsp page operates in the same way as
its network counterpart: it tests a complete query going through apache/tomcat/web module/database and
uploading to the client. If everything is working properly, it returns an "OK". We recommend running this
test on machines with access to the databases (mtas and surveys, for instance).
Usage: a session token associated with an operator login must be passed as an argument in order to log in
remotely (see the tip in Automatic monitoring via Neolane scripts [page 27]).
24 | © Neolane 2013
Production procedures
For example:
The operator name and login need to be previously configured in the Neolane client console with database
rights.
Frequency: this is a test that uses very little bandwidth. It can therefore be run fairly often (e.g. once every
ten seconds).
Neolane v6.1 - Production Guide - Production procedures | 25
Neolane
/nl/jsp/monitor.jsp
This is a test to check that an operator can access the Neolane server via a web page; the same web page
as the one accessed via the client console menus. We recommend including it in your surveillance tools
(Tivoli, Nagios, etc.).
Usage: a session token associated with an operator login which lets you connect to the instance needs to
be used as an argument (see the tip in Automatic monitoring via Neolane scripts [page 27]).
The operator and their login needs to be configured previously in the Neolane client console with the
appropriate database rights and restrictions.
Frequency: this is a full server test and doesn't need be run often (it can be carried out once every ten
minutes, for example).
/nl/jsp/soaprouter.jsp
This jsp represents the point of entry of Neolane application APIs. It can therefore provide detailed monitoring
of the application. It can also be used to monitor Neolane web services. It is used in our monitoring scripts,
but note that it is for power users only.
Monitoring based on deployment types
Neolane enables various deployment configurations (for more on this, refer to Deployment types). This
section details the various automatic monitoring techniques to be applied depending on your type of installation.
Deployment type
Monitoring
Stand-alone
n
/r/test and /nl/jsp/monitor.jsp on the Neolane server
Standard
n
/r/test and /nl/jsp/ping.jsp on the frontal servers
n
/nl/jsp/monitor.jsp on the application server
n
/r/test and /nl/jsp/ping.jsp on the frontal servers
n
/r/test and /nl/jsp/monitor.jsp on the application server
Enterprise
26 | © Neolane 2013
Production procedures
Deployment type
Monitoring
Mid-sourcing
n
/nl/jsp/monitor.jsp on the application server
Automatic monitoring via Neolane scripts
Neolane provides an instance monitoring tool (netreport) that lets you send a report by email regarding the
detected anomalies.
Required elements
The following pre-installation precautions are required for automatic monitoring:
n
n
n
n
You must have the netreport.tgz (Linux installation) or netreport.zip (Windows installation) files,
We strongly advise you not to install monitoring on the machine to be monitored,
it must be installed on a machine with a JRE or a JDK,
in Linux, the machine to be monitored must have the bc package.
Installation procedure
The installation procedure is as follows:
1 In the console, create a new operator if necessary (the 'monitoring' user already exists), but do not assign
any rights.
Run archive extraction.
2
3 Read the readme file.
4 Update the netconf.xml configuration file.
5 Update the netreport.bat (Windows) or netreport.sh (Linux) file.
Configuring the netconf.xml file
The XML configuration file contains the following elements:
n
n
n
n
'Properties' element [page 28],
'Instance' element [page 29]
'Host' element [page 29],
Sub-elements [page 30].
Here is a configuration example:
<?xml version="1.0" encoding="ISO-8859-1"?>
<netconf>
<properties mailServer="mail.neolane.net" mailFrom="mail@adobe.com"
recipientList="recipient@adobe.com">
<nightMode start="00:00 am" end="07:00 am"/>
Neolane v6.1 - Production Guide - Production procedures | 27
Neolane
<buildRange minimum="7829" maximum="8180"/>
<buildRange minimum="8300" maximum="8400"/>
<sla/>
</properties>
<instance name="dev61" recipientList="mail@mail.com,mail2@mail.com">
<host name="dev61rd.domain.com" alias="dev61rd" sessiontoken="monitoring"
criticalLevel="1" filter="wkf;new">
<ncs instance="dev61rd" url="/nl/jsp/soaprouter.jsp"
includeDead="false" isSecure="false"/>
<redir url="/r/test"/>
<http url="/nl/jsp/ping.jsp"/>
</host>
<host name="dev61trk.domain.com" alias="dev61trk" sessiontoken="monitoring"
criticalLevel="0" filter="wkf;new">
<ncs instance="dev61rd" url="/nl/jsp/soaprouter.jsp"
includeDead="true" isSecure="false"/>
</host>
</instance>
<host name="dev-test" alias="dev-test" sessiontoken="monitoring" criticalLevel="2">
<ncs instance="dev" url="/nl/jsp/soaprouter.jsp" includeDead="false"/>
</host>
</netconf>
Note:
You can specify various configurations by adding a suffix to the netconf.xml file, for example,
netconf-dev.xml, netconf-prod.xml, etc. Then specify the configuration to use for executing the netreport
in the netreport.bat or netreport.sh files by adding $JAVA_HOME/bin/java netreport dev or
@%JAVA_HOME%\bin\java netreport prod for example.
Warning:
For the monitoring operator to work, the machine that the netreport is executed on must be in a security
zone that is in sessionTokenOnly mode. If no trusted IP mask has been specified for this operator, the
security zone must also be in allowEmptyPassword and allowUserPassword mode.
'Properties' element
This element is used to populate the configuration of emails, i.e.
n
n
n
n
n
mailServer: SMTP server used to send emails (e.g.: smtp.neolane.net).
mailFrom: email address of the report sender (e.g.: <monitoring@neolane.net>).
recipientList: the list of email addresses of monitoring recipients. Addresses must be separated by
commas (no spaces).
'night' mode (optional) is used to make sure no emails are sent between the time specified. Instead,
the data is consolidated and an email concerning the night's activity is sent after the end time (7:00 by
default).
The buildRange sub-element (optional) lets you specify a minimum and maximum build number . An
error will be generated for all machines whose build number does not fall into this range.
<buildRange minimum="0000" maximum="9999"/>
n
You can add an <sla/> (optional) sub-element in the properties element. A log file will be generated
everytime the netreport is executed. The name of the file contains the configuration name and the time
and date, for example dev_06_12_13_16_47_05.tmp. The file contains the following information: instance
name, machine name, severity level, (0 to 3, from least critical to most critical), date (timestamp format),
time elapsed (in milliseconds) between the query and the response, service used (http, ncs, ncsex, redir).
This information is separated by tabulation marks and line breaks at the end of each service.
Note:
The persistHtmlFile attribute with the value "true" on the <property> element is used to record the
latest monitoring status in the file netreport.html. This file is saved in the installation directory.
28 | © Neolane 2013
Production procedures
'Instance' element
This element lets you regroup several machines (hosts) into the same instance. The instance names appear
in the first part of the monitoring email. You can click on the name of an instance to access detail regarding
each machine.
<instance name="instanceName" recipientList="mail@mail.com,mail2@mail.com">
<host name="dev61rd.domain.com" ...>
...
</host>
<host name="dev61trk.domain.com" ...>
...
</host>
</instance>
n
n
name: instance name that will appear in the first part of the email.
recipientList (optional): lets you send a monitoring report regarding a particular instance by email.
'Host' element
This element configures the monitoring of a given server on the host, i.e.
n
n
n
name: name of the machine to be monitored.
alias (optional): name of the monitored machine as it will appear in the report.
sessionToken: provides login authentication via an authorized session token.
To configure the session token, select the monitoring operator in the Neolane console. In the Access
rights tab, specify the IP addresses of the machines authorized to monitor this instance. You will then
be able to connect to the monitoring page from those machines using the monitoring identifier and
without needing to specify a password.
n
criticalLevel (optional): lets you sort errors to be displayed by level of severity. Possible values are '0'
(all levels displayed), '1' (only high and critical errors displayed) and '2' (only critical errors displayed). If
this attribute is not provided, all error levels are displayed.
Neolane v6.1 - Production Guide - Production procedures | 29
Neolane
n
filter (optional): lets you exclude certain workflow errors, for example filter="wkf;wkf1". Workflow
labels must be serarated by commas.
Sub-elements
n
n
n
n
tcp: checks if the server is up or down. You must enter a port number.
http: checks that the Web server exists (application server is operational).
ncs: checks the processes on the instance entered in the 'instance' attribute (workflow errors, memory
usage, etc.). The includead (mandatory) gives you the option of displaying dead processes ('true' or
'false' values).
redir: checks the tracking.
In most cases, only the ncs and redir sub-elements can be kept.
In any case, certain nodes can be overloaded in the sub-elements (e.g., the node port=75 to overload the
port used for the http, ncs or redir connection).
ncs instance="clap40" url="/nl/jsp/soaprouter.jsp" includeDead="false" port="80"/
In the ncs, redir and http sub-elements, you can add the isSecure attribute (optional) to choose whether
or not to use the https protocol ('true' or 'false' values). If this attribute is not provided, the http protocol is
used.
Configuring the netreport.bat or netreport.sh file
To configure it, edit this file and indicate which directory the JRE or JDK is installed in.
Launching monitoring
To launch monitoring, execute the netreport.bat or netreport.sh file at regular intervals via a script. A
report is sent after the first execution, and then only in the event of a change of status.
Testing monitoring
To test the monitoring, execute the netreport.bat or netreport.sh file.
An email is sent to the recipients specified in the recipientList of the netconf.xml file.
Usual commands
This section lists the usual commands in Neolane.
The command nlserver is the input command for the whole Neolane application.
This command has the following syntax: nlserver <command> <arguments>
The parameter <command> corresponds to the module.
Note:
n
n
In any case, you can add the -noconsole argument to delete comments displayed once the modules
are started.
Conversely, you can add the argument -verbose to display more information.
Monitoring commands
Note:
To list all modules, you need to use the nlserver pdump command
You can add the parameter -who to list the connections in progress (database and application).
nlserver pdump -who
11:00:39 >
Application server for Neolane Version 6.0.0 (build 6749) from 26/07/2011
web@default (9984) - 50.1 Mo
watchdog (2273) - 6.6 Mo
syslogd@default (9931) - 7.0 Mo
30 | © Neolane 2013
Production procedures
trackinglogd@default (9985) - 45.6 Mo
mta@test (9986) - 9.6 Mo
wfserver@test (9987) - 8.8 Mo
Connections -----------------------------------------------------Last Access
IP
Instance
Login
21/07/2011 10:00:25 127.0.0.1
default
formation_fr|tracking
26/07/2011 10:07:44 127.0.0.1
default
internal|monitoring
Connection pool -------------------------------------------------Datasource
Server
Provider
Login
default xxxxx
myserver
myprovider
test400
Another useful command is nlserver monitor. It lists the monitoring XML file (obtained in the Neolane
client or via the monitor.jsp web page).
You can add the parameter -missing to list the absent modules (error in modules, modules shut down, etc.)
nlserver monitor -missing
11:00:39 >
Application server for Neolane Version 6.0.0 (build 6749) from 26/07/2011
inMail@test
mta@test
wfserver@test
This corresponds to the modules with automatic startup but which have not been launched.
Module launch commands
The syntax to launch modules will still have the following format:
nlserver start <ModuleName>@<INSTANCE>
nlserver stop <NameModule>@<INSTANCE>
Note:
<INSTANCE> corresponds to the name of the instance as entered in the configuration files, or default for
mono-instance modules.
Shut down services
To stop Neolane services, use one of the following commands:
n
If you have root or administrator access:
n
In Linux:
/etc/init.d/nlserver6 stop
n
In Windows:
net stop nlserver6
n
If not, then in the Neolane account:
nlserver shutdown
Restart services
Similarly, in order to restart Neolane you can use one of the following commands:
n
If you have root or administrator access:
n
In Linux:
/etc/init.d/nlserver6 start
n
In Windows:
Neolane v6.1 - Production Guide - Production procedures | 31
Neolane
net start nlserver6
n
Otherwise, in the Neolane account: nlserver watchdog -svc -noconsole
The config command
The config command lets you manage server configuration, including the reconfiguration of the database
connection.
Use the config command of the nlserver executable file with the -setdblogin parameter.
nlserver config -setdblogin:<[dbms:]account[:database][/password]@server>
nlserver config -setdblogin:PostgreSQL:neolane:test6@dbserver
Enter the password.
To change the internal password: nlserver config -internalpassword
Warning:
To log on with the Internal identifier, you need to have defined a password beforehand. For more on this,
refer to the Installation guide.
Note:
n
n
n
In general, instead of modifying the configuration files by hand, you can use the config command
To get the list of parameters, use the -? parameter: nlserver config -?
In the case of an Oracle database, you must not specify the account. The syntax will be as follows:
nlserver config -setdblogin:Oracle:test6@dbserver
32 | © Neolane 2013
CHAPTER 3
Data processing
Table of Contents
Backup . . . . . . . . . . . . . . . . .
Physical files . . . . . . . . . . . . . .
Database . . . . . . . . . . . . . . .
Restoration . . . . . . . . . . . . . . .
Duplicating environments . . . . . . . . . .
Introduction . . . . . . . . . . . . . .
Implementation . . . . . . . . . . . . .
Description of the database Cleanup workflow . . .
Introduction . . . . . . . . . . . . . .
Configuration . . . . . . . . . . . . .
Tasks carried out by the Database cleanup workflow
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
33
33
34
34
34
34
35
38
38
38
40
Backup
Backing up is essential in order to avoid losing data in the event of a problem (whether physical or
system-related) on a machine.
Data is stored in two separate locations:
n
n
physical files are stored in the Neolane directories,
other data is stored in the database.
Most of the data is in the database. This represents 99% of the information to be backed up.
Physical files
Files are divided into several categories:
n
Configuration files, located in nl6/conf
n
These enable you to reconfigure Neolane very quickly.
Redirection files nl6/var/<instanceName>/redir
n
These are on the tracking (often called 'frontal') servers, and include all previous campaign
redirections. They are still used by previous campaigns.
Log files: nl6/var/<instanceName>/log
n
These can be used to trace problems.
The pURL configuration files (Neolane Microsites): nl6/var/<instanceName>/relay
Neolane v6.1 - Production Guide - Data processing | 33
Neolane
These files, which are essential to microsite operation, define the way pURLS are relayed to the Web
applications.
Important:
The directories to be backed up are therefore:
n
nl6/conf
n
nl6/var/<instanceName>/redir (for each instance)
n
nl6/var/<instanceName>/log (optional)
n
nl6/var/<instanceName>/relay (optional)
Database
The database contains all of the information displayed in the Neolane rich client console, as well as all the
line-of-business data.
Warning:
It is essential to back up the database.
Your hosting company, and their database administrators in particular, are responsible for this operation.
Restoration
On a clean server, the restoration procedure is as follows:
n
n
n
n
n
n
on an installed and configured operating system (networks),
install third-party applications: Web server, JDK (if necessary),
installation of Neolane binaries with the same build as the source system,
copy configuration files, tracking logs and redirection files,
create and rebuild the database,
start Neolane.
Note:
For more information, refer to the Installation Guide.
Duplicating environments
Introduction
Overview
Using Neolane requires installing and configuring one or more environments: development, test, pre-production,
production, etc.
Each environment contains a Neolane instance and each Neolane instance is linked to one or more databases.
The application server can execute one or more processes: almost all of these have direct access to the
instance database.
This section details the processes to be applied to duplicate a Neolane environment, i.e. to restore a source
environment to a target environment, resulting in two identical work environments.
To do this, apply the following steps:
1 create a copy of the databases on all instances in the source environment,
2 restore these copies on all instances of the target environment,
34 | © Neolane 2013
Data processing
3 Run the nms:freezeInstance.js cauterization script on the target environment before starting it up.
This process doesn't impact the servers and their configuration.
Note:
In the context of Neolane, a cauterization combines actions that let you stop all processes interacting
with the outside: logs, tracking, deliveries, campaign workflows, etc.
This step is necessary to avoid delivering messages several times (once from the nominal environment
and one from the duplicated environment).
Warning:
One environment can contain several instances. Each Neolane instance is subjected to a license contract.
Check your license agreement to see how many environments you can have.
The procedure below lets you transfer an environment without impacting the number of environments
and instances you have installed.
Before you start
Warning:
We strongly recommend running a full backup of the databases for all instances of the source and target
environments before starting the transfer process. This way if a problem occurs, you will be able to restore
the backups and return to your initial configuration.
In order for this process to work, the source and target environments must have the same number of
instances, the same purpose (marketing instance, delivery instance) and similar configurations. The technical
configuration must comply with software prerequisites. The same components must be installed on both
environments.
Implementation
Transfer procedure
This section will help you understand the steps required for transferring a source environment to a target
environment via a case study: our aim here is to restore a production environment (prod instance) to a
development environment (dev instance) to work in a context that is as close as possible to the 'live' platform.
The following steps must be performed with great care: some processes may still be in progress when the
source environment databases are copied. Cauterization (step 3 below) prevents messages from being sent
twice and maintains data consistency.
Warning:
n
n
The following procedure is valid in PostgreSQL language. If the SQL language is different (Oracle, MySQL,
etc.), the SQL queries must be adapted.
The commands below apply within the context of a prod instance and a dev instance under PostgreSQL.
Step 1 - Run a backup of the source environment data (prod)
Copy the databases
Start by copying all databases of the source environment. The operation depends on the database engine
and is the responsibility of the database administrator.
Under PostgreSQL, the command is:
pg_dump mydatabase > mydatabase.sql
Step 2 - Export the target environment configuration (dev)
Most configuration elements are different for each environment: external accounts (mid-sourcing, routing,
etc.), technical options (platform name, DatabaseId, email addresses and default URLs, etc.).
Neolane v6.1 - Production Guide - Data processing | 35
Neolane
Before saving the source database on the target database, you need to export the target environment (dev)
configuration. To do this, export the content of these two tables: xtkoption and nmsextaccount.
This export lets you keep the dev configuration and only refresh dev data (workflows, templates, Web
applications, recipients, etc.).
To do this, perform a package export for the following two elements:
n
n
Export the xtk:option table into an 'options_dev.xml' file, without the records with the following internal
names: 'WdbcTimeZone', 'NmsServer_LastPostUpgrade' and 'NmsBroadcast_RegexRules'.
In an 'extaccount_dev.xml' file, export the nms:extAccount table for all records whose ID is not 0 (@id
<> 0).
Check that the number of exported options/accounts is equal to the number of lines to export in each file.
Note:
The number of lines to export in a package export is 1000 lines. If the number of options or external accounts
is more than 1000, you must carry out several exports.
For more information, refer to the Export packages section.
Step 3 - Stop the target environment (dev)
You need to stop Neolane processes on all servers of the target environment. This operation depends on
your operating system.
You can stop all processes, or only those that write to the database.
To stop all processes, use the following commands:
n
In Windows:
net stop nlserver6
n
In Linux:
/etc/init.d/nlserver6 stop
Use the following command to check that all processes are stopped:
nlserver pdump
Note:
In Windows, the webmdl process can still be active without impacting other operations.
You can also check that no system processes remain running.
To do this, use the following process:
n
n
In Windows: open the Task manager and check that there are no nlserver.exe processes.
In Linux: run the ps aux | grep nlserver command and check that there are no nlserver processes.
Step 4 - Restore the databases in the target environment (dev)
To restore the source databases onto the target environment, use the following command:
psql mydatabase < mydatabase.sql
Step 5 - Cauterize the target environment (dev)
To avoid malfunctions, the processes linked to delivery sending and workflow execution must not be executed
automatically when the target environment is activated.
To do this, run the following command:
nlserver javascript nms:freezeInstance.js -instance:<dev> -arg:run
Step 6 - Check cauterization
1 Check that the only deliverypart is the one whose ID is set to 0:
SELECT * FROM neolane.nmsdeliverypart;
36 | © Neolane 2013
Data processing
2 Check that the delivery status update is correct:
SELECT iState, count(*) FROM neolane.nmsdelivery GROUP BY iState;
3 Check that the workflow status update is correct:
SELECT iState, count(*) FROM neolane.xtkworkflow GROUP BY iState;
SELECT iStatus, count(*) FROM neolane.xtkworkflow GROUP BY iStatus;
Step 7 - Restart the target environment Web process (dev)
On the target environment, re-start the Neolane processes for all servers.
Note:
Before re-starting Adobe Campaign on the dev environment, you can apply an additional safety procedure:
start the web module only.
To do this, edit your instance's configuration file (config-dev.xml), then add the "_" character before the
autoStart="true" options for each module (mta, stat, etc.).
Run the following command to start the Web process:
nlserver start web
Use the following command to check that only the web process has started:
nlserver pdump
Check that access to the client console functions.
Step 8 - Import options and external accounts into the target environment (dev)
Warning:
Warning: only the web process should be started at this step. If this is not the case, stop other running
processes before continuting.
Above all, check the values of several lines of the files before importing (for example: 'NmsTracking_Pointer'
for the options table and the delivery or mid-sourcing accounts for the external account table).
To import the configuration from the target environment database (dev):
1 Open the admin console of the database and purge the external accounts (table nms:extAccount) whose
2
ID is not 0 (@id <> 0).
In the Neolane console, import the options_dev.xml package previously created via the import package
functionality.
Check that the options have indeed been updated in the Administration > Platform > Options node.
3 In the Neolane console, import the extaccount_dev.xml previously created via the import package
fonctionality.
Check that external databases have indeed been imported in the Administration > Platform > External
accounts.
Step 9 - Restart all processes and change users (dev)
To start the Neolane processes, use the following commands:
n
In Windows:
net start nlserver6
n
In Linux:
/etc/init.d/nlserver6 start
Use the following command to check that the processes are started:
nlserver pdump
Neolane v6.1 - Production Guide - Data processing | 37
Neolane
Change users to find the users that already existed on the dev platform.
Description of the database Cleanup workflow
Introduction
The Database cleanup workflow accessible via the Administration > Production > Technical workflows
node, lets you delete obsolete data to avoid exponential growth of the database. The workflow is triggered
automatically without user intervention.
Configuration
The database cleanup is configured on two levels: in the workflow scheduler and in the deployment wizard.
The scheduler
Note:
For more on the scheduler, view the relevant section of the Workflows guide.
By default, the Database cleanup workflow is configured to start daily at 4AM. The scheduler lets you
change the workflow triggering frequency. The following frequencies are available:
n
n
n
Several times a day
Daily
Weekly
38 | © Neolane 2013
Data processing
n
Once
Warning:
In order for the Database cleanup workflow to start at the date and time defined in the scheduler, the
workflow engine (wfserver) must be started. If this isn't the case, database cleansing won't take place until
next time the workflow engine is started.
Neolane v6.1 - Production Guide - Data processing | 39
Neolane
Deployment wizard
The Deployment wizard, accessed via the Tools > Advanced menu, lets you configure how long data
is saved for. Values are expressed in days. If these values aren't altered, the workflow will use the default
values.
The fields of the Purge of data window coincide with the following options. These are used by some of the
tasks executed by the Database cleanup workflow:
n
n
n
n
n
n
n
Consolidated tracking: NmsCleanup_TrackingStatPurgeDelay (refer to Cleanup of tracking logs
[page 45])
Delivery logs: NmsCleanup_BroadLogPurgeDelay (refer to Cleanup of delivery logs [page 45])
Tracking logs: NmsCleanup_TrackingLogPurgeDelay (refer to Cleanup of tracking logs [page 45])
Deleted deliveries: NmsCleanup_RecycledDeliveryPurgeDelay (refer to Cleanup of deliveries to be
deleted or recycled [page 41])
Import rejects: NmsCleanup_RejectsPurgeDelay (refer to Cleanup of rejects generated by imports
[page 44])
Visitor profiles: NmsCleanup_VisitorPurgeDelay (refer to Cleanup of visitors [page 45])
Offer propositions: NmsCleanup_PropositionPurgeDelay (refer to Cleanup of propositions [page 47])
Note:
The Offer propositions field is only available when the Interaction module is installed.
n
n
Events: NmsCleanup_EventPurgeDelay (refer to Cleansing expired events [page 48])
Archived events: NmsCleanup_EventHistoPurgeDelay (refer to Cleansing expired events [page 48])
Note:
The Events and Archived events fields are only available if the Message Center module is installed.
All tasks executed by the Database cleanup workflow are described in the following section.
Tasks carried out by the Database cleanup workflow
At the date and time defined in the workflow scheduler (refer to The scheduler [page 38]), the workflow
engine starts the database cleanup process. The Database cleanup connects to the database and executes
the tasks in the sequence shown below.
40 | © Neolane 2013
Data processing
Warning:
If one of these tasks fails, the following ones will not be executed.
Note:
SQL queries with a LIMIT attribute will be executed repeatedly until all information is processed.
Lists to delete cleanup
The first task executed by the Database cleanup workflow deletes all groups with the deleteStatus !=
0 attribute from the NmsGroup. Records linked to these groups and which exist in other tables are also
deleted.
1 Lists to be deleted are recovered using the following SQL query:
SELECT iGroupId, sLabel, iType FROM
tsExpirationDate <= GetDate()
NmsGroup WHERE
iDeleteStatus <> 0 OR
2 Each list has several links to other tables. All of these links are deleted in bulk using the following query:
DELETE FROM $(relatedTable) WHERE iGroupId=$(l) IN (SELECT iGroupId FROM $(relatedTable)
WHERE iGroupId=$(l) LIMIT 5000)
3
where $(relatedTable) is a table related to NmsGroup and $(l) is the list identifier.
When the list is a 'List' type list, the associated table is deleted using the following query:
DROP TABLE grp$(l)
4 Every Select type list recovered by the operation is deleted using the following query:
DELETE FROM NmsGroup WHERE iGroupId=$(l)
where $(l) is the list identifier.
Cleanup of deliveries to be deleted or recycled
This task purges all deliveries to be deleted or recycled.
1 The Database cleanup workflow selects all deliveries for which the deleteStatus field has the value
Yes or Recycled and whose delete date is earlier than the period defined in the Deleted deliveries
(NmsCleanup_RecycledDeliveryPurgeDelay) field of the deployment wizard. For more on this, refer
to Deployment wizard [page 40]. This period is calculated in relation to the current server date.
For each mid-sourcing server, the task selects the list of deliveries to be deleted.
2
3 The Database cleanup workflow deletes delivery logs, attachments, mirror page information and all
4
other related data.
Before deleting the delivery for good, the workflow purges linked information from the following tables:
n
In the delivery exclusion table (NmsDlvExclusion), the following query is used:
DELETE FROM NmsDlvExclusion WHERE iDeliveryId=$(l)
n
where $(l) is the identifier of the delivery.
In the coupon table (NmsCouponValue), the following query is used (with mass-deletions):
DELETE FROM NmsCouponValue WHERE iMessageId IN (SELECT iMessageId FROM NmsCouponValue
WHERE EXISTS (SELECT B.iBroadLogId FROM $(BroadLogTableName) B WHERE B.iDeliveryId
= $(l) AND B.iBroadLogId = iMessageId ) LIMIT 5000)
n
n
n
where $(l) is the identifier of the delivery.
In the delivery log tables (NmsBroadlogXxx), mass-deletions are executed in batches of 10,000
records.
In the offer proposition tables (NmsPropositionXxx), mass-deletions are executed in batches of
10,000 records.
In the tracking log tables (NmsTrackinglogXxx), mass-deletions are executed in batches of 5,000
records.
Neolane v6.1 - Production Guide - Data processing | 41
Neolane
n
n
n
n
n
In the delivery fragment table (NmsDeliveryPart), mass-deletions are executed in batches of 5,000
records. This table contains personalization information on the remaining messages to be delivered.
In the mirror page data fragment table (NmsMirrorPageInfo), mass-deletions are executed in
batches of 5,000 records. This table contains personalization information on all messages used for
generating mirror pages.
In the mirror page search table (NmsMirrorPageSearch), mass-deletions are executed in batches
of 5,000 records. This table is a search index which provides access to personalization information
stored in the NmsMirrorPageInfo table.
In the batch process log table (XtkJobLog), mass-deletions are executed in batches of 5,000 records.
This table contains the log of deliveries to be deleted.
In the delivery URL tracking table (NmsTrackingUrl), the following query is used:
DELETE FROM NmsTrackingUrl WHERE iDeliveryId=$(l)
where $(l) is the identifier of the delivery.
This table contains the URLs found in the deliveries to be deleted to enable their tracking.
5 The delivery is deleted from the delivery table (NmsDelivery):
DELETE FROM NmsDelivery WHERE iDeliveryId = $(l)
where $(l) is the identifier of the delivery.
Deliveries using mid-sourcing
The Database cleanup workflow also deletes deliveries on the mid-sourcing server(s).
1 To do this, the workflow checks that each delivery is inactive (based on its status). If a delivery is active,
it will be stopped before it is deleted. The check is carried out by executing the following query:
SELECT iState FROM NmsDelivery WHERE iDeliveryId = $(l) AND iState <> 100;
2
where $(l) is the identifier of the delivery.
If the value of the status is Start pending, In progress, Recovery pending, Recovery in progress,
Pause requested, Pause in progress, or Paused (values 51, 55, 61, 62, 71, 72, 75), the delivery is
stopped and the task purges the linked information.
Cleanup of expired deliveries
This task stops deliveries whose validity period has expired.
1 The Database cleanup workflow creates the list of deliveries which have expired. This list includes all
expired deliveries with a status other than Finished, as well as recently stopped deliveries with over
10,000 non-processed messages. The following query is used:
SELECT iDeliveryId, iState FROM NmsDelivery WHERE iDeleteStatus=0 AND iIsModel=0 AND
iDeliveryMode=1 AND ( (iState >= 51 AND iState < 85 AND tsValidity IS NOT NULL AND
tsValidity < $(currentDate) ) OR (iState = 85 AND DateMinusDays(15) < tsLastModified
AND iToDeliver - iProcessed >= 10000 ))
2
where delivery mode 1 matches the Mass delivery mode, state 51 matches the Start pending
state, state 85 matches the Stopped state, and the highest number of delivery logs mass-updated
on the delivery server equals 10,000.
The workflow then includes the list of recently expired deliveries which use mid-sourcing. Deliveries for
which no delivery logs have yet been recovered via the mid-sourcing server are excluded.
The following query is used:
SELECT iDeliveryId, tsValidity, iMidRemoteId, mData FROM NmsDelivery WHERE (iDeliveryMode
= 4 AND (iState = 85 OR iState = 95) AND tsValidity IS NOT NULL AND (tsValidity <
SubDays(GetDate() , 15) OR tsValidity < $(DateOfLastLogPullUp)) AND tsLastModified >
SubDays(GetDate() , 15))
3 The following query is used to detect whether or not the external account is still active, for filtering
deliveries by date:
SELECT iExtAccountId FROM NmsExtAccount WHERE iActive<>0 AND sName=$(providerName)
4 In the list of expired deliveries, delivery logs whose status is Pending, switch to Delivery cancelled,
and all deliveries in this list switch to Finished.
42 | © Neolane 2013
Data processing
The following queries are used:
UPDATE $(BroadLogTableName) SET tsLastModified=$(curdate), iStatus=7, iMsgId=$(bl)
WHERE iDeliveryId=$(dl) AND iStatus=6
where $(curdate) is the current date of the database server, $(bl) is the identifier of the delivery
logs message, $(dl) is the delivery identifier, delivery status 6 matches the Pending status and
delivery status 7 matches the Delivery cancelled status.
UPDATE NmsDelivery SET iState = 95, tsLastModified = $(curdate), tsBroadEnd = tsValidity
WHERE iDeliveryId = $(dl)
5
where delivery state 95 matches the Finished status, and $(dl) is the identifier of the delivery.
All fragments (deliveryParts) of obsolete deliveries are deleted and all obsolete fragments of notification
deliveries in progress are deleted. Mass-deletion is used for both these tasks.
The following queries are used:
DELETE FROM NmsDeliveryPart WHERE iDeliveryPartId IN (SELECT iDeliveryPartId FROM
NmsDeliveryPart WHERE iDeliveryId IN (SELECT iDeliveryId FROM NmsDelivery WHERE iState=95
OR iState=85) LIMIT 5000)
DELETE FROM NmsDeliveryPart WHERE iDeliveryPartId IN (SELECT iDeliveryPartId FROM
NmsDeliveryPart WHERE tsValidity < $(curDate) LIMIT 500000)
where delivery state 95 matches the Finished status, delivery state 85 matches the Stopped
status, and $(curDate) is the current server date.
Cleanup of mirror pages
This task deletes the web resources (mirror pages) used by deliveries.
1 First of all, the list of deliveries to be purged is recovered using the following query:
SELECT iDeliveryId, iNeedMirrorPage FROM NmsDelivery WHERE iWebResPurged = 0 AND
tsWebValidity IS NOT NULL AND tsWebValidity < $(curdate)"
where $(curDate) is the current server date.
2 The NmsMirrorPageInfo table is then purged, if necessary using the identifier of the previously
recovered delivery. Mass-deletion is used to generate the following queries:
DELETE FROM NmsMirrorPageInfo WHERE iMirrorPageInfoId IN (SELECT iMirrorPageInfoId FROM
NmsMirrorPageInfo WHERE iDeliveryId = $(dl)) LIMIT 5000)
DELETE FROM NmsMirrorPageSearch WHERE iMessageId IN (SELECT iMessageId FROM
NmsMirrorPageSearch WHERE iDeliveryId = $(dl)) LIMIT 5000)
where $(dl) is the identifier of the delivery.
An entry is then added to the delivery log.
3
4 Purged deliveries are then identified, to avoid having to reprocess them later. The following query is
executed:
UPDATE NmsDelivery SET iWebResPurged = 1 WHERE iDeliveryId IN ($(strIn))
where $(strIn) is the list of delivery identifiers.
Cleanup of work tables
This task deletes from the database, all work tables which match deliveries whose status is Being edited,
Stopped or Deleted.
1 The list of tables with names beginning with wkDlv_ is recovered first with the following query
(postgresql):
SELECT relname FROM pg_class WHERE relname LIKE Lower('wkDlv_') ESCAPE E'\\\\' AND
relkind IN ('r','v') AND pg_get_userbyid(relowner)<>'postgres'
2 The tables used by workflows in progress are then excluded. To do this, the list of deliveries in progress
is recovered using the following query:
SELECT iDeliveryId FROM NmsDelivery WHERE iDeliveryId<>0 AND iDeleteStatus=0 AND iState
NOT IN (0,85,100);
Neolane v6.1 - Production Guide - Data processing | 43
Neolane
3
where 0 is the value which matches the Being edited delivery status, 85 matches the Stopped status
and 100 matches the Deleted status.
Tables that are no longer used will be deleted using the following query:
DROP TABLE wkDlv_15487_1;
Cleanup of rejects generated by imports
This step lets you delete records for which all data wasn't processed during import.
1 Mass-deletion is carried out on the XtkReject table with the following query:
DELETE FROM XtkReject WHERE iRejectId IN (SELECT iRejectId FROM XtkReject WHERE tsLog
< $(curDate)) LIMIT $(l))
2
where $(curDate) is the current server date from which we subtract the period defined for the
NmsCleanup_RejectsPurgeDelay option (refer to Deployment wizard [page 40]) and $(l) is the
maximum number of records to be mass deleted.
All orphan rejects are then deleted using the following query:
DELETE FROM XtkReject WHERE iJobId NOT IN (SELECT iJobId FROM XtkJob)
Cleanup of workflow instances
This task purges each workflow instance using its identifer (lWorkflowId) and history (lHistory). It
deletes inactive tables by running the worktable cleanup task again.
1 To recover the list of workflows to be deleted, the following query is used:
SELECT iWorkflowId, iHistory FROM XtkWorkflow WHERE iWorkflowId<>0
2 This query generates the list of workflows which will be used to delete all linked logs, finished tasks and
finished events, using the following queries:
DELETE FROM XtkWorkflowLog WHERE iWorkflowId=$(lworkflow) AND tsLog <
DateMinusDays($(lhistory))
DELETE FROM XtkWorkflowTask WHERE iWorkflowId=$(lworkflow) AND iStatus<>0 AND
tsCompletion < DateMinusDays($(lhistory))
DELETE FROM XtkWorkflowEvent WHERE iWorkflowId=$(l) AND iStatus>2 AND tsProcessing <
DateMinusDays($(lHistory))
where $(lworkflow) is the identifier of the workflow and $(lhistory) is the identifier of the history.
3 All unused tables are deleted. For this purpose, all tables are collected thanks to a wkf% type mask
using the following query (postgresql):
SELECT relname FROM pg_class WHERE relname LIKE Lower('wkf%') ESCAPE E'\\\\' AND relkind
IN ('r','v') AND pg_get_userbyid(relowner)<>'postgres'
4 Then all tables used by a pending workflow instance are excluded. The list of active workflows is recovered
using the following query:
SELECT iWorkflowId FROM XtkWorkflow WHERE iWorkflowId<>0 AND iState<>20
5 Each workflow identifier is then recovered to find the name of the tables used by workflows in progress.
These names are excluded from the list of previously recovered tables.
6 "incremental query" type activity history tables are excluded using the following queries:
SELECT relname FROM pg_class WHERE relname LIKE Lower('wkfhisto%') ESCAPE E'\\\\' AND
relkind IN ('r','v') AND pg_get_userbyid(relowner)<>'postgres'
SELECT iWorkflowId FROM XtkWorkflow WHERE iWorkflowId IN ($(strCondition))
where $(strcondition) is the list of tables which match the wkfhisto% mask.
7 The remaining tables are deleted using the following query:
DROP TABLE wkf15487_12;
44 | © Neolane 2013
Data processing
Cleanup of workflow logins
This task deletes workflow logins using the following query:
DELETE FROM XtkWorkflowLogin WHERE iWorkflowId NOT IN (SELECT iWorkflowId FROM XtkWorkflow)
Cleanup of orphan work tables
This task deletes orphan work tables linked to groups. The NmsGroup table stores the groups to be cleansed
(with a type different from 0). The prefix of the table names is grp. To identify the groups to be cleansed,
the following query is used:
SELECT iGroupId FROM NmsGroup WHERE iType>0"
Cleanup of visitors
This task deletes obsolete records from the visitor table using mass-deletion. Obsolete records are those for
which the last modification is earlier than the conservation period defined in the deployment wizard (refer
to Deployment wizard [page 40]). The following query is used:
DELETE FROM NmsVisitor WHERE iVisitorId IN (SELECT iVisitorId FROM NmsVisitor WHERE
iRecipientId = 0 AND tsLastModified < $(tsDate) LIMIT 5000)
where $(tsDate) is the current server date, from which we subtract the period defined for the
NmsCleanup_VisitorPurgeDelay option.
Cleanup of NPAI
This task lets you delete records which match valid addresses from the NmsAddress table. The following
query is used to perform mass-deletion:
DELETE FROM NmsAddress WHERE iAddressId IN (SELECT iAddressId FROM NmsAddress WHERE iStatus=2
AND tsLastModified < $(tsDate1) AND tsLastModified >= $(tsDate2) LIMIT 5000)
where status 2 matches the Valid status, $(tsDate1) is the current server date, and $(tsDate2)
matches the NmsCleanup_LastCleanup option.
Cleanup of subscriptions
This task purges all subscriptions deleted by the user from the NmsSubscription table, using mass-deletion.
The following query is used:
DELETE FROM NmsSubscription WHERE iDeleteStatus <>0
Cleanup of tracking logs
This task deletes obsolete records from the tracking and webtracking log tables. Obsolete records are those
which are earlier than the conservation period defined in the deployment wizard (refer to Deployment wizard
[page 40]).
1 First, the list of tracking log tables is recovered using the following query:
SELECT distinct(sTrackingLogSchema) FROM NmsDeliveryMapping WHERE sTrackingLogSchema
IS NOT NULL;
2 Mass-deletion is used to purge all tables in the list of previously recovered tables. The following query is
used:
DELETE FROM XtkTrackingLogRcp WHERE iTrackingLogId IN (SELECT iTrackingLogId FROM
XtkTrackingLogRcp WHERE tsLog < $(tsDate) LIMIT 5000)
3
where $(tsDate) is the current server date from which we subtract the period defined for the
NmsCleanup_TrackingLogPurgeDelay option.
The tracking statistics table is purged using mass-deletion. The following query is used:
DELETE FROM NmsTrackingStats WHERE iTrackingStatsId IN (SELECT iTrackingStatsId FROM
NmsTrackingStats WHERE tsStart < $(tsDate) LIMIT 5000)
where $(tsDate) is the current server date from which we subtract the period defined for the
NmsCleanup_TrackingStatPurgeDelay option.
Cleanup of delivery logs
This task lets you purge the delivery logs stored in various tables.
Neolane v6.1 - Production Guide - Data processing | 45
Neolane
1 For this purpose, the list of delivery log schemas is recovered using the following query:
SELECT distinct(sBroadLogSchema) FROM NmsDeliveryMapping WHERE sBroadLogSchema IS NOT
NULL UNION SELECT distinct(sBroadLogExclSchema) FROM NmsDeliveryMapping WHERE
sBroadLogExclSchema IS NOT NULL
2 When using mid-sourcing, the NmsBroadLogMid table isn't referenced in delivery mappings. The
3
nms:broadLogMid schema is added to the list recovered by the previous query.
The Database cleanup workflow then purges obsolete data from previously recovered tables. The
following query is used:
DELETE FROM $(tableName) WHERE iBroadLogId IN (SELECT iBroadLogId FROM $(tableName)
WHERE tsLastModified < $(option) LIMIT 5000)
4
where $(tableName) is the name of each table in the list of schemas, and $(option) is the date
defined for the NmsCleanup_BroadLogPurgeDelay option (refer to Deployment wizard [page 40]).
Finally, the workflow checks whether the NmsProviderMsgId table exists. If so, all obsolete data is
deleted using the following query:
DELETE FROM NmsProviderMsgId WHERE iBroadLogId IN (SELECT iBroadLogId FROM
NmsProviderMsgId WHERE tsCreated < $(option) LIMIT 5000)
where $(option) matches the date defined for the NmsCleanup_BroadLogPurgeDelay option
(refer to Deployment wizard [page 40]).
Cleanup of the NmsEmailErrorStat table
This task cleanses the NmsEmailErrorStat table. The main program (coalesceErrors) defines two dates:
n
n
Start date: date of the next process which matches the NmsLastErrorStatCoalesce option or the
most recent date in the table.
End date: current server date.
If the start date is greater than or equal to the end date, no process will take place. In this case, the
coalesceUpToDate message appears.
If the start date is earlier than the end date, the NmsEmailErrorStat table is cleansed.
The total number of errors in the NmsEmailErrorStat table, between the start and end dates, is recovered
using the following query:
"SELECT COUNT(*) FROM NmsEmailErrorStat WHERE tsDate>= $(start) AND tsDate< $(end)"
where $end and $start are the start and end dates defined previously.
If the total is greater than 0:
1 The following query is executed in order to keep only errors beyond a certain threshold (which equals
20):
"SELECT iMXIP, iPublicId, SUM(iTotalConnections), SUM(iTotalErrors), SUM(iMessageErrors),
SUM(iAbortedConnections), SUM(iFailedConnections), SUM(iRefusedConnections),
SUM(iTimeoutConnections) FROM NmsEmailErrorStat WHERE tsDate>=$(start ) AND tsDate<$(end
) GROUP BY iMXIP, iPublicId HAVING SUM(iTotalErrors) >= 20"
2 The coalescingErrors message is displayed.
3 A new connection is created to delete all errors which occurred between the start and end dates. The
following query is used:
"DELETE FROM NmsEmailErrorStat WHERE tsDate>=$(start) AND tsDate<$(end)"
4 Each error is saved in the NmsEmailErrorStat table using the following query:
"INSERT INTO NmsEmailErrorStat(iMXIP, iPublicId, tsDate, iTotalConnections, iTotalErrors,
iTimeoutConnections, iRefusedConnections, iAbortedConnections, iFailedConnections,
iMessageErrors) VALUES($(lmxip ), $(lpublicId ), $(tsstart ), $(lconnections ),
$(lconnectionErrors ),$(ltimeoutConnections ), $(lrefusedConnections ),
$(labortedConnections ), $(lfailedConnections ), $(lmessageErrors))"
where each variable matches a value recovered by the previous query.
5 The start variable is updated with the values of the previous process to finish the loop.
The loop and the task stop.
46 | © Neolane 2013
Data processing
Cleanups are executed on the NmsEmailError and cleanupNmsMxDomain tables.
Cleanup of the NmsEmailError table
The following query is used:
DELETE FROM NmsEmailError WHERE iMXIP NOT IN (SELECT DISTINCT iMXIP FROM NmsEmailErrorStat)
This query deletes all lines without linked records in the NmsEmailErrorStat from the NmsEmailError
table.
Cleanup of the NmsMxDomain table
The following query is used:
DELETE FROM NmsMxDomain WHERE iMXIP NOT IN (SELECT DISTINCT iMXIP FROM NmsEmailErrorStat)
This query deletes all lines without a linked record in the NmsEmailErrorStat table from the NmsMxDomain
table.
Cleanup of propositions
If the Interaction module is installed, this task is executed to purge the NmsPropositionXxx tables.
The list of propositions tables is recovered and mass-deletion is carried out on each one, using the following
query:
DELETE FROM NmsPropositionXxx WHERE iPropositionId IN (SELECT iPropositionId FROM
NmsPropositionXxx WHERE tsLastModified < $(option) LIMIT 5000)
where $(option) is the date defined for the NmsCleanup_PropositionPurgeDelay option (refer to
Deployment wizard [page 40]).
Cleanup of simulation tables
This task cleanses orphan simulation tables (that are no longer linked to an offer simulation or a delivery
simulation).
1 To recover the list of simulations that require cleanup, the following query is used:
SELECT iSimulationId FROM NmsSimulation WHERE iSimulationId<>0
2 The name of the tables to delete is made up of the wkSimu_ prefix followed by the identifier of the
simulation (for instance: wkSimu_456831_aggr):
DROP TABLE wkSimu_456831_aggr
Statistics update
If the XtkCleanup_NoStats option does not exist or if its value is 0 (i.e. the option is disabled), this task
is executed to update the statistics in the database. If the value of the option is 1 (i.e. the option is activated),
statistics updating isn't executed. This task calls up a database procedure.
Subscription cleanup (NMAC)
This task deletes any subscriptions related to deleted services or mobile applications.
1 To recover the list of broadlog schemas, the following query is used:
SELECT distinct(sBroadLogSchema) FROM NmsDeliveryMapping WHERE sBroadLogSchema IS NOT
NULL
2 The task then recovers the names of the tables linked to the appSubscription link and deletes these
tables.
Cleansing session information
This task cleanses information from the sessionInfo table, the following query is used:
DELETE FROM XtkSessionInfo WHERE tsexpiration < $(curdate)
Neolane v6.1 - Production Guide - Data processing | 47
Neolane
Cleansing expired events
This task cleanses the events received and stored on the execution instances and the events archived on a
control instance.
Cleansing reactions
This task cleanses the reactions (NmsRemaMatchRcp table) in which the hypotheses have themselves
been deleted.
48 | © Neolane 2013
CHAPTER 4
Updating Neolane
Table of Contents
Upgrade . . . . . . . . . . . . . . . .
In Windows . . . . . . . . . . . . . .
In Linux . . . . . . . . . . . . . . .
Resolving upgrade conflicts . . . . . . . . .
Informing client consoles of the available update .
Example of the Unicode switch of an existing instance
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
49
50
51
52
53
54
This section presents the procedure to be applied to upgrade Neolane, client side and server side,
and describes the switching to Unicode of an existing instance.
The upgrade must be applied to all servers where Neolane is installed.
Neolane is based on several processes executed on the server side which you will need to manipulate
during updates, in particular:
n
n
n
Application server (nlserver web)
Delivery server (nlserver mta)
Redirection server (webmdl)
Note:
For more on the various Neolane processes, refer to the Installation guide.
When using the Power Booster or Power Cluster type architecture, you must apply this process to
all Power Booster/Cluster servers.
If the new version involves an alteration of the database structure, we recommend restarting the
servers in the following order:
1 Application server (nlserver web),
2 Redirection server (webmdl),
3 Delivery server (nlserver mta).
Upgrade
Note:
In the following commands, the XXXX characters represent the Neolane build number.
Neolane v6.1 - Production Guide - Updating Neolane | 49
Neolane
Warning:
We strongly recommend making a database backup on each instance before updating. For more information,
refer to Backup [page 33].
In Windows
To update Neolane in a new version when delivering a new build, the following procedure should be applied
in Windows:
n
n
n
n
Shut down services [page 50],
Upgrade the Neolane server application [page 50],
Synchronizing resources [page 50],
Restart services [page 51].
To find out how to update the client console, refer to Client console availability.
Shut down services
In order to replace all files with the new version, you need to shut down all instances of the nlserver service.
1 Shut down the following services:
n
Web services (IIS):
n
iisreset /stop
Neolane service: net stop nlserver6
Warning:
You also need to make sure the redirection server (webmdl) is stopped, so that the nlsrvmod.dll file
used by IIS can be replaced with the new version.
2 Check that no tasks are active by running the nlserver pdump command. The following should come
up:
C:\Program Files\Neolane\Neolane v6\bin>nlserver pdump
11:33:39 > Application Server for Neolane Version 6.00 (build 6745) dated 11/07/2011
No tasks
You can use the Windows Task Manager to make sure all processes are stopped.
Upgrade the Neolane server application
To run the upgrade file, apply the following steps:
1 Run setup.exe.
To download this file, go to the Neolane Support page (http://support.neolane.net/) via the Download
Center link.
Select the installation mode: choose Update or repair
2
3 Click Next.
4 Click Finish.
5
The installation program then copies the new files.
Once the operation is complete, click Finish.
Synchronizing resources
Use the following command line
nlserver config -postupgrade -allinstances
To carry out the following operations:
n
n
n
Synchronize resources,
update schemas,
update the database.
50 | © Neolane 2013
Updating Neolane
Note:
This operation should be performed only once, and only on an (nlserver web) application server.
Then check whether the synchronization has generated errors or warnings. For more on this, refer to Resolving
upgrade conflicts [page 52].
Tip:
To synchronize one database only, use the following command: nlserver config -postupgrade
-instance:<instance_name>.
Restart services
The services to be restarted are:
n
Web services (IIS):
n
iisreset /start
Neolane service: net start nlserver6
In Linux
To update Neolane in a new version when a new build is delivered, the procedure for Linux is as follows:
n
n
n
Obtain updated packages [page 51],
Performing an update [page 51],
Rebooting the web server [page 51].
To find out how to update the client console, refer to Client console availability.
Obtain updated packages
Start by recovering both updated packages of Neolane.
The files are:
n
nlthirdparty6-XXXX-linux-2.6-amd64.deb or nlthirdparty6-XXXX-0.x86_64.rpm
n
nlserver6-XXXX-linux-2.6-amd64.deb or nlserver6-XXXX-0.x86_64.rpm
Performing an update
n
RPM based distribution (RedHat, SuSe)
To install them, execute as root:
$rpm -Uvh nlthirdparty6-XXXX-0.x86_64.rpm nlserver6-XXXX-0.x86_64.rpm
n
DEB based distribution (Debian)
To install them, execute as root:
dpkg -i nlthirdparty6-XXXX-linux-2.6-amd64.deb nlserver6-XXXX-linux-2.6-amd64.deb
Note:
Resources are synchronized automatically, however you need to make sure no errors occurred. For more on
this, refer to Resolving upgrade conflicts [page 52].
Rebooting the web server
You must shut down Apache for the new library to become applicable.
To do this, execute the following command:
/etc/init.d/apache stop
Neolane v6.1 - Production Guide - Updating Neolane | 51
Neolane
Warning:
n
n
Your script might be called httpd instead of apache.
You MUST execute this command until you obtain the following reply:
httpd (no pid file) not running
This operation is required in order for Apache to apply the new library.
Then restart Apache:
/etc/init.d/apache start
Resolving upgrade conflicts
During resource synchronization, the postupgrade command enables you to detect whether synchronization
has generated errors or warnings.
View the synchronization result
There are two ways of viewing the synchronization result:
n
In the command-line interface, errors are materialized by a triple chevron >>> and synchronization is
stopped automatically. Warnings are materialized by a double chevron >> and must be resolved once
synchronization is complete. At the end of the postupgrade, a summary is displayed in the command
prompt. It can look like this:
2013-04-09 07:48:39.749Z
00002E7A
1
info
log
=========Summary
of the update==========
2013-04-09 07:48:39.749Z
00002E7A
1
info
log
neolanetest
instance, 6 warning(s) and 0 error(s) during the update.
2013-04-09 07:48:39.749Z
00002E7A
1
warning log
The document
with identifier 'mobileAppDeliveryFeedback' and type 'xtk:report' is in conflict with
the new version.
2013-04-09 07:48:39.749Z
00002E7A
1
warning log
The document
with identifier 'opensByUserAgent' and type 'xtk:report' is in conflict with the new
version.
2013-04-09 07:48:39.750Z
00002E7A
1
warning log
The document
with identifier 'deliveryValidation' and type 'nms:webApp' is in conflict with the new
version.
2013-04-09 07:48:39.750Z
00002E7A
1
warning log
Document of
identifier 'nms:includeView' and type 'xtk:srcSchema' updated in the database and found
in the file system. You will have to merge the two versions manually.
n
If the warning concerns a conflict of resources, user attention is required to resolve it.
The postupgrade_<server version number>_<time of postupgrade>.log log file contains the
synchronization result. It is available by default in the following directory: <installation
directory/var/<instance/postupgrade. Errors and warnings are indicated by the error and warning
attributes.
Resolving conflicts
To resolve conflicts, apply the following process:
1 In the Neolane tree, go to Administration>Configuration>Package management>Edit conflicts.
2 Select the conflict you want to resolve in the list.
There are three ways to resolve a conflict:
n
n
n
Declare as resolved: requires user intervention beforehand.
Accept the new version: recommended if the resources provided with Neolane have not been changed
by the user.
Keep the current version: means that the update is rejected.
Warning:
If you select this resolution mode, you may not benefit from corrections in the new version.
52 | © Neolane 2013
Updating Neolane
If you chose to resolve the conflict manually, proceed as follows:
1 In the lower section of the window, search for the _conflict_ string to locate the entities with conflicts.
The entity installed with the new version contains the new argument, the entity that matches the previous
version contains the cus argument.
2 Delete the version you don't want to keep. Delete the _conflict_argument_ string of the entity you
are keeping.
3 Go to the conflict you have resolved. Click the Actions icon and select Declare as resolved.
4 Save your changes: the conflict is now resolved.
Informing client consoles of the available update
In Windows
On the machine where the (nlserver web) Neolane application server is installed, download and copy the
file
setup-client-6.XXXX.exe
in [path of the application]\datakit\nl\en"\jsp
The next time client consoles are connected, a window will inform users about the availability of an update
and offer them the possibility of downloading and installing it.
Note:
Make sure the IIS_XPG user has the appropriate read rights for this installation file and refer to the
Installation Guide for more information.
In Linux
On the machine where the Neolane application server (nlserver web) is installed, retrieve the following
package:
setup-client-6.XXXX.exe
Neolane v6.1 - Production Guide - Updating Neolane | 53
Neolane
and copy it, saving as /usr/local/neolane/nl6/datakit/nl/en/jsp:
cp setup-client-6.XXXX.exe /usr/local/neolane/nl6/datakit/nl/en/jsp
The next time client consoles are connected, a window will inform users about the availability of an update
and offer them the possibility of downloading and installing it.
Note:
Make sure the Apache user has the appropriate read rights for this installation file and refer to the Installation
Guide for more information.
Example of the Unicode switch of an existing instance
For an existing prod instance in Linux/PostgreSQL, the steps for switching to unicode are as follows:
1 Stop the processes writing to the database:
su - neolane
nlserver shutdown
2 Dump the database:
su - postgres
pg_dump mydatabase > mydatabase.sql
3 Create a Unicode database:
createdb -E UNICODE mydatabase_unicode
4 Restore the database:
psql mydatabase_unicode < mydatabase.sql
5 Update the option indicating that the database is Unicode:
psql mydatabase_unicode
update XtkOption set sStringValue = 'u'||sStringValue where sName='XtkDatabaseId' and
sStringValue not like 'u%';
6 On the tracking servers:
su - neolane
cd nl6/conf
vi config-prod.xml
Add the u character in front of the value relating to the database identifier (databaseId):
<web>
<redirection databaseId="u7F0000010554364C" trackingPassword="myPassword="/>
</web>
7 On servers calling the database:
su - neolane
cd nl6/conf
vi config-prod.xml
Modify the database reference:
<dataSource name="default">
<dbcnx encrypted="1"
login="neolane:mydatabase_unicode" password="xxxx="
provider="postgresql" server="yyyy"/>
</dataSource>
8 Reboot all the machines:
54 | © Neolane 2013
Updating Neolane
/etc/init.d/apache stop
/etc/init.d/nlserver6 stop
/etc/init.d/nlserver6 start
/etc/init.d/apache start
9 Confirm the switch. To do this, connect via the Neolane console and:
n
n
check that the data is displayed correctly, in particular the accentuated characters:
launch a delivery and check that the tracking retrieval works.
Neolane v6.1 - Production Guide - Updating Neolane | 55
Neolane
56 | Neolane v6.1 - Production Guide
CHAPTER 5
Database maintenance
Table of Contents
List of tables to maintain . . .
Neolane tables . . . . . .
Customer tables . . . . .
Types of maintenance . . . .
Application maintenance . . .
Technical maintenance . . .
RDBMS Specific recommendations
PostgreSQL . . . . . . .
Oracle . . . . . . . . .
DB2 . . . . . . . . . .
SQL Server . . . . . . .
MySQL . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
57
58
60
61
61
61
62
62
67
68
68
76
Neolane is a highly transactional system (OLTP database). This means that the underlying database
will be frequently updated, leading to a degradation of performance over time. To avoid this type
of issue, regular database maintenance is necessary.
Warning:
A database will only perform optimally if it is maintained on a regular basis. The automatic
maintenance offered by some RDBMSs isn't sufficient and does not replace in-depth maintenance.
This isn't specific to Neolane but concerns all relational database transactional management systems.
This chapter details how to create maintenance plans to suit your needs.
Warning:
The procedures described in this document are recommendations. The person in charge of database
maintenance plans is the database administrator. Please contact this person and not Neolane in
case of problems.
List of tables to maintain
The list of tables to maintain depends on your version of Neolane, how you use it and on the
datamodel configuration.
The following list contains only the tables most subject to fragmentation. The impacts are as follows:
Neolane v6.1 - Production Guide - Database maintenance | 57
Neolane
n
n
overconsumption of disk space, thus affecting database access,
indexes which have not been regularly updated, which slows down query performance.
Neolane tables
From Neolane 4.05 upward
Table name
Size
Main type of activity
Comments
NmsDelivery
Small
Updates
There is one record per delivery
action. A single record can be
updated several times to reflect
delivery progress, so indexes on
this table tend to fragment rapidly.
NmsDeliveryPart
Medium
Insertions, updates, deletions
Work table which records are inserted into during delivery preparation. They are then updated
during delivery and finally deleted
once the delivery is complete.
This table tends to fragment rapidly even though its average size
is fairly limited.
NmsMirrorPageInfo
Large
Insertions, deletions
This table contains the information needed to generate personalized mirror pages. It contains a
memo (CLOB) field, and as such
will tend to be very large. The
volume is directly proportional to
the history of mirror pages kept.
NmsDeliveryStat
Medium
Insertions, updates, deletions
This table contains statistics on
the delivery process. Its records
are regularly updated.
NmsAddress
Medium
Updates, insertions
This table contains information
on email addresses. It is frequently updated as part of the
quarantine process (records are
created at the first delivery error,
updated when the counters
change and deleted once delivery
is successful).
XtkWorkflow
Small
Updates
There is one record per workflow
instance, so very few records.
However the table it is regularly
updated to reflect status and
progress.
XtkWorkflowTask
Small
Insertions, updates, deletions
Each execution of a workflow
activity leads to the creation of a
record in this table. The purge
mechanism deletes them once
they are expired.
XtkWorkflowEvent
Small
Insertions, updates, deletions
Each transition activated between
tasks in a workflow leads to the
creation of a record in this table.
The purge mechanism deletes
them once they are expired.
58 | © Neolane 2013
Database maintenance
Table name
Size
Main type of activity
Comments
XtkWorkflowJob
Very small
Insertions, updates, deletions
This table is specific to the workflow engine. It enables the sending of commands to workflows
(Start, Stop, Pause, for instance).
Although it is small, this table is
taken into account during the
purge of transactional tables
linked to workflows.
From versions 4.05 and 5.00 upward
Table name
Size
Main type of activity
Comments
NmsBroadLog
Largest
Insertions, updates, deletions
This is the largest table in the
system. There is one record per
message sent, and these records
are inserted, updated to track the
delivery status, and deleted when
the history is purged.
NmsTrackingLog
Large volume
Insertions, deletions
Tracking logs are inserted and
deleted when the history is
purged, but they are not updated.
From Neolane 5.10 upward
Table name
Size
Main type of activity
Comments
NmsBroadlogMsg
Small
Updates
This table contains information
used for qualifying SMTP errors.
It is fairly small, but will be
massively updated, so indexes on
this table tend to fragment rapidly.
NmsEmailErrorStat
Medium
Insertions, updates, deletions
This table contains the aggregates on SMTP errors sorted by
domain. It initially contains detailed information which is aggregated by the cleanup task once it
becomes outdated.
NmsBroadLogMid (on a midsourcing instance)
Large
Insertions, updates, deletions
Only when the Neolane 5.10 (or
later) instance is used as a midsourcing instance. This is one of
the largest table in the database.
There is one record per message
sent, and these records are inserted, updated to track the delivery
status, and deleted when the
history is purged. When using
mid-sourcing, the recommendation is to limit the history (usually
less than two months), so this
table remains reasonable in terms
of size (less than 30 Go for 60
million rows, data+index), but it
is very important to rebuild it
from time to time.
Neolane v6.1 - Production Guide - Database maintenance | 59
Neolane
Table name
Size
Main type of activity
Comments
NmsBroadLogRcp (when the Nm- Large
sRecipient table is used)
Insertions, updates, deletions
This is the largest table in the
system. There is one record per
message sent, and these records
are inserted, updated to track the
delivery status, and deleted when
the history is purged. Note that
in 5.10, this table is smaller than
the equivalent in 4.05 (NmsBroadLog) since the SMTP message
text is factorized in the NmsBroadLogMsg table in the 5.10 version.
However, it remains essential to
re-index this table regularly
(every other week to start with),
and completely rebuild it from
time to time (once a month, or
when performance is impacted).
YyyBroadLogXxx (when an extern- Large
al recipient table is used)
Insertions, updates, deletions
Same as NmsBroadLogRcp but
with an external recipient table.
Please adapt Yyy and Xxx with
the values in your delivery mapping.
NmsTrackingLogRcp (when the
NmsRecipient table is used)
Large
Insertions, deletions
Tracking logs are inserted and
deleted when the history is
purged, but they are not updated.
The volume depends on the
length of data retention.
YyyTrackingLogXxx (when the
external recipient table is used)
Large
Insertions, deletions
Same as NmsTrackingLogRcp but
with an external recipient table.
Please adapt Yyy and Xxx with
the values used in your delivery
mapping.
Table name
Size
Main activity
Comments
NmsMobileApp
Very small volume
Insertions, updates, deletions
Tables that include mobile applications and their configuration.
NmsAppSubscriptionRcp
Large volume
Insertions, updates
Table that includes the identifiers
of mobile devices (addresses)
used to send the notification
(similar to a recipient table).
NmsBroadLogAppSubRcp
Large volume
Insertions, updates, deletions
Similar to the other broadlog
tables, but with the NmsappSubscriptionRcp instead of NmsRecipient.
NmsTrackingLogAppSubRcp
Large volume
Insertions, deletions
Similar to the other trackingLog
tables, but with the NmsappSubscriptionRcp table instead of NmsRecipient.
XtkSessionInfo
Small volume
Insertions, deletions
Table that includes user sessions.
The number of insertions and
deletions is very important.
From version 6.1
Customer tables
In addition to the list above, tables containing created by customers (which do not exist in the Neolane
datamodel) during platform setup can also be subject to fragmentation, specially if they are frequently
updated during data loading or synchronization procedures. These tables can be part of the default Neolane
data model (for instance NmsRecipient). In this case, it is up to the administrator of the Neolane platform
to conduct an audit of its specific database model to find these custom tables. These tables aren't necessarily
mentioned explicitly in our maintenance procedures.
60 | © Neolane 2013
Database maintenance
Types of maintenance
Application maintenance
Neolane provides an out-of-the-box workflow which lets you schedule certain database maintenance tasks.
This feature is the database cleanup workflow, which carries out the following tasks:
n
n
n
deletion of expired records,
deletion of orphaned records and status reinitialization for expired objects,
updating the database statistics.
Warning:
Please note that the cleanup task deals mostly with application level maintenance, not with RDBMS level
maintenance (with the exception of statistics update). However maintenance operations will be required on
the database. Just because the cleanup runs successfully does not mean that the database is optimally tuned.
Technical maintenance
The database cleanup workflow does not include any database maintenance tool: it is up to you to organize
maintenance. To do this, you can either:
n
n
ask your DBA to set up database maintenance without using the tools available in the Neolane platform,
use the Neolane workflow engine to schedule and track these maintenance activities.
These maintenance procedures must be carried out on a regular basis and should include the following:
n
n
re-index frequently updated tables,
compact/rebuild the tables to avoid fragmentation.
Maintenance scheduling
The difficulty is finding appropriate slots for performing these maintenance activities, since they can heavily
impact the database performance while running or even block the application (due to locking).
These tasks are typically run once a week during a period of low activity that does not collide with backups,
data reloading or aggregate calculation. Some systems which are highly solicited require more frequent
maintenance.
More in-depth maintenance, such as full table rebuilds, can be performed once a month, preferably with
applications fully stopped since the system is unusable anyway.
Rebuilding a table
Several strategies are available:
Operations
Description
Benefits
Drawbacks
Online defragmentation
Most database engines provide
defragmentation methods.
Simply use the database defragmentation method. These methods typically take care of integrity
issues by locking the data during
defragmentation.
Depending on the database,
these defragmentation methods
can be provided as an RDBMS
option (Oracle) and aren't always
the most efficient way of dealing
with larger tables.
Dump and restore
Dump the table to a file, delete This is the easiest way to defragthe table in the database and re- ment a table. Also the only solustore from the dump.
tion when the database is almost
full.
Since the table is deleted and recreated, the application cannot
be left online, even in read only
mode (the table is not available
during the restore phase).
Neolane v6.1 - Production Guide - Database maintenance | 61
Neolane
Operations
Description
Benefits
Drawbacks
Duplicate, rename and drop
This creates a copy of a table and
its indexes, then drops the existing one and renames the copy to
take its place.
this method is faster than the first
approach since it generates less
IOs (no copy as a file and read
from this file).
Requires twice the amount of
space.
All active processes writing to the
table during the process must be
stopped. However, reading processes will not be affected, since
the table is swapped at the last
moment once rebuilt.
RDBMS Specific recommendations
To help you set up maintenance plans, this section lists some recommendations/best practices adapted to
the various RDBMS engines that Neolane supports. However, these are only recommendations. It is up to
you to adapt them to your needs, in keeping with your internal procedure and constraints. Your DBA is the
person who should ultimately be responsible for these plans.
PostgreSQL
Detecting large tables
1 You can add the following view to your database:
create or replace view uvSpace
as
SELECT c1.relname AS tablename, c2.relname AS indexname, c2.relpages * 8 / 1024 AS
size_mbytes, c2.relfilenode AS filename, 0 AS row_count
FROM pg_class c1, pg_class c2, pg_index i
WHERE c1.oid = i.indrelid AND i.indexrelid = c2.oid
UNION
SELECT pg_class.relname AS tablename, NULL::"unknown" AS indexname, pg_class.relpages
* 8 / 1024 AS size_mbytes, pg_class.relfilenode AS filename, cast(pg_class.reltuples
as integer) AS row_count
FROM pg_class
WHERE pg_class.relkind = 'r'::"char"
ORDER BY 3 DESC, 1, 2 DESC;
2 Running the following command allows you to spot large tables and indexes:
select * from uvSpace;
Simple maintenance
Under PostgreSQL, the typical commands you can use are vacuum full and reindex.
Here is a typical example of an SQL maintenance plan to be executed on a regular basis using these two
commands:
vacuum full nmsdelivery;
reindex table nmsdelivery;
vacuum full nmsdeliverystat;
reindex table nmsdeliverystat;
vacuum full xtkworkflow;
reindex table xtkworkflow;
vacuum full xtkworkflowevent;
reindex table xtkworkflowevent;
vacuum full xtkworkflowjob;
reindex table xtkworkflowjob;
vacuum full xtkworkflowlog;
reindex table xtkworkflowlog;
vacuum full xtkworkflowtask;
62 | © Neolane 2013
Database maintenance
reindex table xtkworkflowtask;
vacuum full xtkjoblog;
reindex table xtkjoblog;
vacuum full xtkjob;
reindex table xtkjob;
vacuum full nmsaddress;
reindex table nmsaddress;
vacuum full nmsdeliverypart;
reindex table nmsdeliverypart;
vacuum full nmsmirrorpageinfo;
reindex table nmsmirrorpageinfo;
Note:
n
n
n
n
n
We recommend starting with smaller tables, this way if the process fails on large tables (where the risk
of failure is highest), at least part of the maintenance has been completed.
Also add the tables specific to your data model which can be subject to significant updates. This can be
the case for NmsRecipient if you have large daily data replication flows.
The vacuum and re-index commands will lock the table, which pauses some processes while
maintenance is carried out.
For very large tables (typically above 5 Gb), vacuum full can become quite inefficient and take a very
long time. We do not recommend using it for the YyyNmsBroadLogXxx table.
This maintenance operation can be implemented by a Neolane workflow, using an SQL activity (for more
on this, refer to the Workflows guide). Make sure you schedule maintenance for a low activity time which
does not collide with your backup window.
Rebuilding a database
PostgreSQL does not provide an easy way of performing an online table rebuild since vacuum full locks
the table, thus preventing regular production. This means that maintenance has to be performed when the
table is not used. You can either:
n
n
perform maintenance when the Neolane platform is stopped,
stop the various Neolane sub-services likely to write in the table being rebuilt (nlserver stop
wfserver instance_name to stop the workflow process).
Here is an example of table defragmentation using specific functions to generate the necessary DDL. The
following SQL lets you create two new functions: GenRebuildTablePart1 and GenRebuildTablePart2,
which can be used to generate the necessary DDL to recreate a table.
n
n
n
The second function then deletes the original table and renames the work table and its indexes.
The second function then deletes the original table and renames the work table and its indexes.
Using two functions instead of one means that if the first one fails, you don't run the risk of deleting the
original table.
-- --------------------------------------------------------------------------- Generate the CREATE TABLE DDL for a table
-- -------------------------------------------------------------------------create or replace function GenTableDDL(text) returns text as $$
declare
vstrTable text;
vrecFld RECORD;
vstrDDL text;
vstrFields text;
vstrNsTable text;
vstrTableSpace text;
begin
vstrTable = lower($1);
vstrDDL = ;
SELECT
pg_catalog.quote_ident(n.nspname) || '.' || pg_catalog.quote_ident(c.relname),
Neolane v6.1 - Production Guide - Database maintenance | 63
Neolane
pg_catalog.quote_ident(t.spcname)
INTO
vstrNsTable, vstrTableSpace
FROM
pg_namespace n, pg_class c left outer join pg_tablespace t on c.reltablespace = t.oid
WHERE
n.oid = c.relnamespace AND
c.relname = vstrTable;
vstrDDL = 'CREATE TABLE ' || vstrNsTable || '_tmp(';
vstrFields = ;
FOR vrecFld IN
SELECT
pg_catalog.quote_ident(a.attname) ||
' ' || t.typname ||
case when t.typname='varchar' then '(' || cast(a.atttypmod-4 as text) || ')'
when t.typname='numeric' then '(' || cast((a.atttypmod-4)/65536 as text) ||
',' || cast((a.atttypmod-4)%65536 as text) || ')'
else end ||
case when a.attnotnull then ' not null' else end ||
case when a.atthasdef then ' default '|| d.adsrc else end as DDL
FROM
pg_type t, pg_class c, pg_attribute a LEFT OUTER JOIN pg_attrdef d ON
d.adrelid=a.attrelid and d.adnum=a.attnum
WHERE
a.attnum > 0 AND
a.attrelid = c.oid AND
t.oid = a.atttypid AND
c.relname = vstrTable
ORDER BY
a.attnum
LOOP
IF vstrFields <> THEN
vstrFields = vstrFields || ',' || chr(10) || ' ';
ELSE
vstrFields = vstrFields || chr(10) || ' ';
END IF;
vstrFields = vstrFields || vrecFld.DDL;
END LOOP;
vstrDDL = vstrDDL || vstrFields || chr(10) || ')';
if vstrTableSpace <> then
vstrDDL = vstrDDL || ' TABLESPACE ' || vstrTableSpace;
end if;
vstrDDL = vstrDDL || ';' || chr(10);
return vstrDDL;
END;
$$ LANGUAGE plpgsql;
-- --------------------------------------------------------------------------- Generate the CREATE INDEX DDL for a table
-- -------------------------------------------------------------------------create or replace function GenIndexDDL(text) returns text as $$
declare
vstrTable text;
vrecIndex RECORD;
vstrDDL text;
viFld integer;
vstrFld text;
begin
vstrTable = lower($1);
vstrDDL = ;
FOR vrecIndex IN
SELECT
i.indkey, i.indisunique,
pg_catalog.quote_ident(c.relname) as tablename,
64 | © Neolane 2013
Database maintenance
pg_catalog.quote_ident(ic.relname) as indexname,
pg_catalog.quote_ident(t.spcname) as tablespace
FROM
pg_class c, pg_index i, pg_class ic left outer join pg_tablespace t on
ic.reltablespace = t.oid
WHERE
i.indexrelid = ic.oid AND
i.indrelid = c.oid AND
c.relname = vstrTable
LOOP
vstrDDL = vstrDDL || 'CREATE ';
if vrecIndex.indisunique then
vstrDDL = vstrDDL || 'UNIQUE ';
end if;
vstrDDL = vstrDDL ||
'INDEX ' ||vrecIndex.indexname || '_tmp ON ' ||
vrecIndex.tablename || '_tmp(';
FOR viFld IN array_lower(vrecIndex.indkey, 1) .. array_upper(vrecIndex.indkey, 1)
LOOP
SELECT pg_catalog.quote_ident(a.attname) INTO vstrFld
FROM
pg_attribute a, pg_class c
WHERE
a.attnum = vrecIndex.indkey[viFld] AND
a.attrelid = c.oid AND c.relname=vstrTable;
vstrDDL = vstrDDL || vstrFld;
if viFld <> array_upper(vrecIndex.indkey, 1) then
vstrDDL = vstrDDL || ', ';
end if;
END LOOP;
vstrDDL = vstrDDL || ')';
if vrecIndex.tablespace <> then
vstrDDL = vstrDDL || 'TABLESPACE ' || vrecIndex.tablespace;
end if;
vstrDDL = vstrDDL || ';' || chr(10);
END LOOP;
return vstrDDL;
END;
$$ LANGUAGE plpgsql;
-- --------------------------------------------------------------------------- Generate the ALTER INDEX RENAME for a table
-- -------------------------------------------------------------------------create or replace function GenRenameIndexDDL(text) returns text as $$
declare
vstrTable text;
vrecIndex RECORD;
vstrDDL text;
begin
vstrTable = lower($1);
vstrDDL = ;
FOR vrecIndex IN
SELECT
pg_catalog.quote_ident(n.nspname) as namespace,
pg_catalog.quote_ident(ic.relname) as indexname
FROM
pg_namespace n, pg_class c, pg_index i, pg_class ic
WHERE
i.indexrelid = ic.oid AND
n.oid = ic.relnamespace AND
i.indrelid = c.oid AND
c.relname = vstrTable
LOOP
Neolane v6.1 - Production Guide - Database maintenance | 65
Neolane
vstrDDL = vstrDDL || 'ALTER INDEX ' || vrecIndex.namespace || '.' || vrecIndex.indexname
||
'_tmp RENAME TO ' || vrecIndex.indexname ||
';' || chr(10);
END LOOP;
return vstrDDL;
END;
$$ LANGUAGE plpgsql;
-- --------------------------------------------------------------------------- Build a copy of a table, with index
-- -------------------------------------------------------------------------create or replace function GenRebuildTablePart1(text) returns text as $$
declare
vstrTable text;
vstrTmp text;
vstrDDL text;
begin
vstrTable = lower($1);
vstrDDL = ;
SELECT GenTableDDL(vstrTable) INTO vstrTmp;
vstrDDL = vstrDDL|| vstrTmp || chr(10);
vstrDDL = vstrDDL|| 'INSERT INTO ' || vstrTable || '_tmp SELECT * FROM ' || vstrTable
|| ';'||chr(10);
SELECT GenIndexDDL(vstrTable) INTO vstrTmp;
vstrDDL = vstrDDL|| vstrTmp || chr(10);
vstrDDL = vstrDDL|| 'VACUUM ANALYSE '|| vstrTable || '_tmp;' ||chr(10);
return vstrDDL;
end;
$$ LANGUAGE plpgsql;
-- --------------------------------------------------------------------------- Drop the original table and rename the copy
-- -------------------------------------------------------------------------create or replace function GenRebuildTablePart2(text) returns text as $$
declare
vstrTable text;
vstrTmp text;
vstrDDL text;
begin
vstrTable = lower($1);
vstrDDL = 'DROP TABLE ' || vstrTable||';'|| chr(10);
vstrDDL = vstrDDL|| 'ALTER TABLE ' || vstrTable || '_tmp RENAME TO ' || vstrTable ||';'||
chr(10);
SELECT GenRenameIndexDDL(vstrTable) INTO vstrTmp;
vstrDDL = vstrDDL|| vstrTmp || chr(10);
return vstrDDL;
end;
$$ LANGUAGE plpgsql;
The following example can be used in a workflow to rebuild the required tables rather than using the
vacuum/rebuild command:
function sqlGetMemo(strSql)
{
var res = sqlSelect("s, m:memo", strSql);
return res.s.m.toString();
}
function RebuildTable(strTable)
66 | © Neolane 2013
Database maintenance
{
// Rebuild a table_tmp
var strSql = sqlGetMemo("select GenRebuildTablePart1('"+strTable+"')");
logInfo("Rebuilding table '"+strTable+"'...");
// logInfo(strSql);
sqlExec(strSql);
// If fails, there is an exception thrown and so we do not delete the original table
strSql = sqlGetMemo("select GenRebuildTablePart2('"+strTable+"')");
logInfo("Swapping table '"+strTable+"'...");
//logInfo(strSql);
sqlExec(strSql);
}
RebuildTable('nmsrecipient');
RebuildTable('nmsrcpgrlrel');
// ... other tables here
Oracle
Compaction of tables and indexes
This can be performed using the shrink space option available with the alter table/index command.
Start by enabling the row movement option of each table as shown below:
alter table nmsdelivery enable row movement;
alter table nmsdeliverystat enable row movement;
alter table xtkworkflow enable row movement;
alter table xtkworkflowevent enable row movement;
alter table xtkworkflowjob enable row movement;
alter table xtkworkflowlog enable row movement;
alter table xtkworkflowtask enable row movement;
alter table xtkjoblog enable row movement;
alter table xtkjob enable row movement;
alter table nmsaddress enable row movement;
alter table nmsdeliverypart enable row movement;
alter table nmsmirrorpageinfo enable row movement;
All versions from Neolane 5.10 up, versions 6 included
alter
alter
alter
alter
table
table
table
table
nmsbroadlogrcp enable row movement;
nmsbroadlogmid enable row movement;
nmsbroadlogmsg enable row movement;
nmsemailerrorstat enable row movement;
Note:
If the enable row movement command fails, you must kill all Oracle sessions using the table. Log in as
system and run the following command:
As system: alter system kill session 'sid,serial#';
You can then compact the tables as shown below:
alter table nmsdelivery shrink space compact;
alter table nmsdelivery shrink space;
alter table nmsdeliverystat shrink space compact;
alter table nmsdeliverystat shrink space;
alter table xtkworkflow shrink space compact;
alter table xtkworkflow shrink space;
alter table xtkworkflowevent shrink space compact;
alter table xtkworkflowevent shrink space;
alter table xtkworkflowjob shrink space compact;
alter table xtkworkflowjob shrink space;
alter table xtkworkflowlog shrink space compact;
Neolane v6.1 - Production Guide - Database maintenance | 67
Neolane
alter table xtkworkflowlog shrink space;
alter table xtkworkflowtask shrink space compact;
alter table xtkworkflowtask shrink space;
alter table xtkjoblog shrink space compact;
alter table xtkjoblog shrink space;
alter table nmsaddress shrink space compact;
alter table nmsaddress shrink space;
alter table nmsmirrorpageinfo shrink space compact;
alter table nmsmirrorpageinfo shrink space;
alter table nmsdeliverypart shrink space compact;
alter table nmsdeliverypart shrink space;
All versions from Neolane 5.10 up, versions 6 included
alter table nmsbroadlogrcp shrink space compact;
alter table nmsbroadlogrcp shrink space;
alter table nmsbroadlogmid shrink space compact;
alter table nmsbroadlogmid shrink space;
alter table nmsbroadlogmsg shrink space compact;
alter table nmsbroadlogmsg shrink space;
alter table nmsemailerrorstat shrink space compact;
alter table nmsemailerrorstat shrink space;
Do the same for indexes as shown below (please adapt the list to suit the needs of your data model):
alter index NMSBROADLOGRCP_LASTMODIFIED shrink space compact;
alter index NMSBROADLOGRCP_LASTMODIFIED shrink space;
...
Note:
If you are using an enterprise version of Oracle, an alternative is to use the alter index xxx rebuild
online command to rebuild indexes.
DB2
Please contact your database administrator to find out about the procedures best suited to your version of
DB2.
SQL Server
Note:
For the SQL Server, you can use the maintenance plan detailed here:
http://ola.hallengren.com/sql-server-index-and-statistics-maintenance.html.
The example shown here concerns SQL Server 2005. If you are using SQL Server 2008, please contact your
database administrator to find out about maintenance procedures for this version.
If you are using Microsoft SQL Server 2005, most maintenance task can be carried out thanks to the wizard.
1 First, connect to Microsoft SQL Server Management Studio, with a login with administrator rights.
2 Go to the Management>Maintenance Plans folder, right-click on it and choose Maintenance Plan
Wizard
68 | © Neolane 2013
Database maintenance
3 Click Next when the first page comes up.
4 Select the type of maintenance plan you want to create (separate schedules for each task or single
schedule for the whole plan), then click the Change... button.
Neolane v6.1 - Production Guide - Database maintenance | 69
Neolane
5 In the Job schedule properties window, select the desired execution settings and click OK, then click
Next.
6 Select the maintenance tasks you want to perform, then click Next.
Note:
We recommend performing at least the maintenance tasks shown below. You may also select the statistics
update task, although it is already carried out by the database cleanup workflow.
70 | © Neolane 2013
Database maintenance
7 In the drop-down list, select the database which you want to run the Database Check Integrity task
on.
8 Select the database and click OK, then click Next.
Neolane v6.1 - Production Guide - Database maintenance | 71
Neolane
9 Configure the maximum size allocated to your database, then click Next.
Note:
If the size of the database exceeds this threshold, the maintenance plan will try to delete unused data
to free up space.
10 Reorganize or rebuild the index:
n
If the index fragmentation rate is between 10% and 40%, a reorganization is recommended.
Choose which databases and objects (tables or views) you want to reorganize, then click Next.
72 | © Neolane 2013
Database maintenance
Note:
Depending on your configuration you can choose either the previously selected tables, or all tables
in your database.
n
If the index fragmentation rate is higher than 40%, a rebuild is recommended.
Select the options you want to apply to the index rebuild task, then click Next.
Note:
The rebuild index process is more constricting in terms of processor use and it locks the database
resources. Tick the Keep index online while reindexing option if you want the index to be available
during the rebuild.
Neolane v6.1 - Production Guide - Database maintenance | 73
Neolane
11 Select the options you want to display in the activity report, then click Next.
12 Check the list of tasks configured for the maintenance plan, then click Finish.
74 | © Neolane 2013
Database maintenance
A summary of the maintenance plan and the statuses of its various steps is displayed.
13 Once the maintenance plan is complete, click Close.
14 In the SQL Server explorer, double-click the Management>Maintenance Plans folder.
Neolane v6.1 - Production Guide - Database maintenance | 75
Neolane
15 Select the Neolane maintenance plan: the various steps are detailed in a workflow.
Note that an object has been created in the SQL Server Agent>Jobs folder. This object lets you start
the maintenance plan. In our example there is only one object since all the maintenance tasks are part
of the same plan.
Warning:
For this object to run, the SQL Server agent must be enabled.
MySQL
Using the OPTIMIZE TABLE command
The OPTIMIZE TABLE command can be used with MySQL 5.0 and 5.1. However, when used on InnoDb
tables (which is the case with Neolane), it has a limited impact:
n
n
it only updates index statistics and frees unused space in the index cluster; it does not rebuild the indexes
or the table (for more on this, refer to http://dev.mysql.com/doc/refman/5.0/en/optimize-table.html).
this command will lock the table, and may therefore be performed during system activity. This means
that maintenance operations must be performed either during low activity, or with the Neolane system
offline (with the associated risk of operations not completing).
76 | © Neolane 2013
Database maintenance
So a first level of maintenance is to run the OPTIMIZE TABLE command on the transactional tables
every other week to start with, and increasing the frequency to every week for heavily used databases,
as shown below:
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
OPTIMIZE
TABLE
TABLE
TABLE
TABLE
TABLE
TABLE
TABLE
TABLE
TABLE
TABLE
TABLE
TABLE
TABLE
TABLE
NmsDelivery;
NmsDeliveryStat;
XtkWorkflow;
XtkWorkflowEvent;
XtkWorkflowJob;
XtkWorkflowLog;
XtkWorkflowTask;
XtkJobLog;
XtkJob;
NmsBroadLogMsg;
NmsEmailErrorStat;
NmsAddress;
NmsDeliveryPart;
NmsMirrorPageInfo;
Defragmenting tables
A full table defragmentation can be achieved by running a dump/restore operation on the concerned tables.
This is performed thanks to shell script commands as shown below (repeat for each table):
/etc/init.d/nlserver6 stop
mysqldump database_name NmsDelivery > NmsDelivery.sql
mysql database_name < NmsDelivery.sql
...
/etc/init.d/nlserver6 start
Note:
Note that this operation must be performed while the Neolane platform is offline, otherwise there is a risk
of data loss. For more information, refer to http://dev.mysql.com/doc/refman/5.0/en/rebuilding-tables.html.
The ALTER TABLE command can also be used:
ALTER TABLE NmsRecipient ENGINE=InnoDB;
ALTER TABLE NmsDeliveryStat ENGINE=InnoDB;
ALTER TABLE XtkWorkflow ENGINE=InnoDB;
ALTER TABLE XtkWorkflowEvent ENGINE=InnoDB;
ALTER TABLE XtkWorkflowJob ENGINE=InnoDB;
ALTER TABLE XtkWorkflowLog ENGINE=InnoDB;
ALTER TABLE XtkWorkflowTask ENGINE=InnoDB;
ALTER TABLE XtkJobLog ENGINE=InnoDB;
ALTER TABLE XtkJob ENGINE=InnoDB;
ALTER TABLE NmsBroadLogMsg ENGINE=InnoDB;
ALTER TABLE NmsEmailErrorStat ENGINE=InnoDB;
ALTER TABLE NmsAddress ENGINE=InnoDB;
ALTER TABLE NmsDeliveryPart ENGINE=InnoDB;
ALTER TABLE NmsMirrorPageInfo ENGINE=InnoDB;
This command can be started with the system still running, but will lead to heavy locking and some operations
may not complete. We recommend using it during low activity periods.
Neolane v6.1 - Production Guide - Database maintenance | 77
Neolane
78 | Neolane v6.1 - Production Guide
CHAPTER 6
Troubleshooting
Table of Contents
Temporary files . . . . . . . . .
Database performances . . . . . .
Configuration . . . . . . . .
Platform configuration . . . . .
Database maintenance . . . . .
Specifics . . . . . . . . . .
Modules and frequent issues . . . .
Log precision . . . . . . . . .
Forwarding of tracking logs . . . . .
OpenOffice . . . . . . . . . .
Email delivery . . . . . . . . .
Workflow execution . . . . . . .
Failure to connect . . . . . . . .
Connection thresholds . . . . . .
Dr Watson in Windows XP (client) . .
Stack trace in Linux . . . . . . .
Encoding of the Oracle database . . .
Reactivating the console update request
Lost password . . . . . . . . .
Abnormal JSP behavior . . . . . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
79
80
80
80
80
81
81
82
83
84
84
84
85
86
86
87
88
88
89
89
This section lists some common errors and explains how to solve them.
Temporary files
If error messages such as the following appear (particularly in delivery logs) when the system is
put into production:
Unable to rename file '/tmp/tmp0000.tmp' to
/usr/local/neolane/nl6/bin/..//var/XXX/mta/86510470.xml ;(errno=18, Invalid
cross-device link) (iRc=-52)
The cause is as follows:
Neolane generates temporary files under /tmp, and then renames them to move them to
/usr/local/neolane/nl6/var. This error occurs when both folders (/tmp and
/usr/local/neolane/nl6/var, which is in fact a symbolic link to /var/nl6) correspond to
different devices. The df command is used for verification.
Neolane v6.1 - Production Guide - Troubleshooting | 79
Neolane
To correct this problem, the temporary files must be generated in the same device as the destination. For
example, by executing:
$ cd ~/nl6/var
$ mkdir tmp
$ vi ~/nl6/customer.sh
and then adding:
export TMPDIR=/usr/local/neolane/nl6/var/tmp
Database performances
Most performance issues are linked to database maintenance. Here are four main leads to help you find the
cause of slow performance:
n
n
n
n
Configuration,
Installation and configuration of the Neolane platform,
database maintenance,
real-time diagnosis.
Configuration
Check that the initial Neolane platform configuration is still valid and if necessary, reassess your customer's
needs in terms of deliverability or database size. We also recommend running a full hardware check (CPU,
RAM, IO system).
Platform configuration
Inappropriate configuration may affect platform performance. We recommend that you check network
configuration, platform deliverability options as well as MTA configuration in the serverConf.xml file.
Database maintenance
Database cleanup task
Please make sure the database cleanup task is operational. To do this, view the log files to see if they contain
any errors.
Maintenance plans
Make sure database maintenance is correctly scheduled and executed. To do this, please contact your
database administrator to learn more about:
n
n
n
their maintenance schedule,
previously executed maintenance plans,
view the script logs.
Warning:
If you are using a mid-sourcing configuration, it is essential for databases to be maintained on a regular
basis. When analyzing a delivery on the marketing platform, the marketing instance sends information to
the mid-sourcing instance. If the process is slowed down, the marketing instance will be impacted.
For more on this, refer to Database maintenance [page 57].
Managing work tables
Please check the number and size of work tables. When they exceed a certain size, database performance
is affected. These tables are created by workflows and deliveries. They remain in the database while workflows
and deliveries are active. To limit the size of work tables, you can carry out the following operations:
80 | © Neolane 2013
Troubleshooting
n
n
n
stop or delete deliveries with the following statuses: Failed, In progress, Ready for delivery, or
Paused.
stop or delete workflows which are paused due to an error,
stop all workflows used for tests which do not contain an End activity and whose status therefore remains
Paused.
Warning:
If the operation takes a long time and frees up a lot of space, this means that in-depth maintenance is
necessary (index rebuilding, etc.). For more on this, refer to Database maintenance [page 57].
Neolane process monitoring
Depending on Neolane installation settings, two tools can be used for platform monitoring:
n
n
the instance production page. For more on this, refer to Manual monitoring [page 15].
the netreport script. For more on this, refer to Automatic monitoring via Neolane scripts [page 27].
Specifics
It may become necessary to run a real-time diagnosis to identify the cause of the issue. Start by checking
the process and platform log files, then monitor database activity while recreating the issue. Pay particular
attention to the following:
n
n
n
the maintenance execution plan,
SQL queries being executed,
whether or not external processes are running at the same time (cleansing, imports, aggregate calculation,
etc.).
Modules and frequent issues
Here is a list of modules impacted by frequent issues:
Module
Execution scope
Troubleshooting
export
Execution of an export process
The operator who scheduled this export needs to re-launch it. Either delta or
full re-launch.
import
Execution of an import process
The operator who scheduled this export needs to re-launch it. Check the database
for duplicates.
inMail
Reading of the bounce mail box
Check this module if bounce mails are no longer forwarded.
mta
Delivers emails
Check this module if mails are no longer being sent.
stat
Maintains MTA connection statist- Check this module if mails are no longer being sent.
ics
runwf
Execution of a workflow instance In case of issues, re-launch the module. If necessary, apply the process to increase log precision. For more on this, refer to Log precision [page 82].
syslogd
Log writing
If there are no logs in the log file: check that the module uses the 6666 port.
For more on this, refer to List of open ports [page 6].
tracking
Consolidating and retrieving
tracking logs
Check this module if tracking logs aren't forwarded anymore.
trackinglogd
Tracking log writing and purging Check this module if tracking logs aren't forwarded anymore and there is no
server
trace of logs in the files on the server. For more on this, refer to Forwarding of
tracking logs [page 83].
watchdog
Startup and monitoring instance
Check this module if no processes are starting.
web
Application server (HTTP and
SOAP)
Check this module if the console and web connections don't work and trigger
an xtk:session type error
Neolane v6.1 - Production Guide - Troubleshooting | 81
Neolane
Module
Execution scope
Troubleshooting
wfserver
Workflow server
Check this module if workflows aren't executing.
Log precision
You can apply this process to all Neolane modules to increase log precision.
It involves relaunching the processes with a higher level of logs.
Warning:
This procedure cancels the services in progress on this module.
Neolane can operate with two levels of log:
1 The Verbose mode is the first level after the standard level. The following command activates it:
nlserver restart <MODULE_NAME> -verbose
Check that the error actually occurred, and then restart the process in the normal way:
nlserver restart <MODULE_NAME> -noconsole
2 The TraceFilter mode, which lets you save the greatest number of logs. It is activated by the following
command:
nlserver stop <MODULE_NAME>; nlserver <MODULE_NAME> -verbose -tracefilter:*
Note:
If you use tracefilter:*, all log types are activated: ncm, rdr, nms, jst, timing, wdbc, ldap, soap, xtk,
xtkquery, session, xtkwriter, network, pop3, inmail
The most useful log types are: wdbc (displays all SQL queries), soap (displays all SOAP calls), ldap
(displays all LDAP queries after authentication), xtkquery (displays the list of all the querydef).
You can use them individually (tracefilter:soap,wdbc for example). You can also activate them all and
choose to exclude certain others: -tracefilter:*,!soap
Check that the error actually occurred, and then restart the process in the normal way:
nlserver restart <MODULE_NAME> -noconsole
Warning:
The logs of these commands are stored in the log file of the module.
Here is an example specific to the Web module. The other modules operate as indicated above.
Before sending this command, check that no job in progress will be affected.
nlserver pdump -who
Next, shut down and restart the module in TraceFilter mode.
nlserver stop web; LD_PRELOAD=libjsig.so nlserver web -tomcat -verbose -tracefilter:*
-tracefile:web_debug@default
Another example:
nlserver stop mta@<INSTANCE_NAME>; nlserver mta -instance:<INSTANCE_NAME> -tracefilter:*
-tracefile:mta_debug@<INSTANCE_NAME>
82 | © Neolane 2013
Troubleshooting
Note:
The Tracefile mode lets you save the logs. In the examples above, the logs are saved in the
var/<INSTANCE-NAME>/mta_debug.log and var/default/web_debug.log files.
Warning:
In Windows, do not add the LD_PRELOAD option. The following command suffices:
nlserver web -tomcat -verbose -tracefilter:*
Check that the problem occurs again, and then restart the module:
nlserver restart web -tomcat -noconsole
All information is available in the file /usr/local/neolane/nl6/var/default/log/web.log.
Forwarding of tracking logs
There can be multiple reasons for tracking logs not being forwarded. We recommend that you check the
following information:
n
n
n
Does the Tracking workflow have errors?
Is the module trackinglogd running on the machine?
Have changes been made? They can trigger a loss of connection to the servers using the tracking alias.
Neolane v6.1 - Production Guide - Troubleshooting | 83
Neolane
OpenOffice
If the error message seems to indicate that the connection to the OpenOffice.org server has failed, we
recommend that you verify the following points:
n
n
Check that OpenOffice is properly configured. To do this, launch it manually once, using an account with
administrator rights in Windows, or using the Neolane user account in Linux
In Windows, check that the Neolane service can interact with the Desktop. To do this, open the list of
Windows services and edit the properties of the Neolane service, and then go to the Connection tab
and select the option Allow System to Interact with Desktop.
For more on this, refer to the relevant section of the Installation guide.
Email delivery
If your email campaigns aren't being sent and their status is Pending, your MTA module might not be started
To check this and to start the module if necessary, apply the following steps:
1 Check that your mta@<instance> modules are launched on your MTA servers.
nlserver pdump
11:00:39 >
Application server for Neolane Version 6.X (build XXXX) of DD/MM/YYYY
[...]
mta@<INSTANCENAME> (9268) - 23.0 Mb
[...]
2 If the MTA is not listed, start it via the following command:
nlserver start mta@<INSTANCENAME>
Note:
Replace <INSTANCENAME> with the name of your instance (production, development, etc.). The instance
name is identified via the configuration files:
[path of application]nl6/conf/config-<INSTANCENAME>.xml
Also check the configuration of traffic management (IPAffinity). For more on this, refer to the Installation
guide.
Workflow execution
If workflows aren't executing and their status is Start in progress, this might mean that the workflow
module isn't launched.
To check this and to start the module if necessary, apply the following steps:
1 Check that your wfserver@<instance> modules are launched on your main application server.
nlserver pdump
11:00:39 >
Application server for Neolane Version 6.X (build XXXX) of DD/MM/YYYY
[...]
wfserver@<INSTANCENAME> (9340) - 11.3 Mb
[...]
2 If the workflow server is not listed, start it using the following command:
nlserver start wfserver@<INSTANCENAME>
84 | © Neolane 2013
Troubleshooting
Note:
Replace <INSTANCENAME> with the name of your instance (production, development, etc.). The instance
name is identified via the configuration files:
[path of application]nl6/conf/config-<INSTANCENAME>.xml
Failure to connect
The reasons for this can be multiple and depend on various contexts.
Check the general configuration of the security zones.
Note:
For more on configuring security zones, refer to the Installation guide.
Check the following information:
1 Network checks
n
Do you have Internet access from your computer?
n
Check that you can connect to websites on the Internet (for example). If you cannot connect, the
problem is on your machine. Contact your system administrator.
Can you connect to the server hosting Neolane via another service?
Connect to the server via SSH or any other means. If this is not possible, your host company has a
problem. Contact their system administrator.
2 Checks on Web server side (IIS/apache/etc.)
n
Does the Web server respond?
n
Connect to the Neolane server access URL using a Web browser: http(s)://<URLSERVER. If it
does not respond, the web server is stopped on the machine. Contact the system administrator of
your host company in order to restart the service.
Has Neolane been correctly integrated?
Log on to the: http(s)://<URLSERVER>/r/test URL. The server should return the following
type of message
<redir status='OK' date='YYYY/MM/DD HH:MM:SS' build='XXXX' host='support.neolane.net'
localHost='server'/>
If you do not obtain this result, check in your Web server configuration that integration is taken into
account. For further information about this procedure, refer to the Installation Guide.
3 Checks on the Neolane side
n
Has the Neolane Web module been launched?
Connect to the following URL: http(s)://<URLSERVER/nl/jsp/logon.jsp
n
If you obtain a Tomcat Java error:
Is the JAVA integration correctly performed? Neolane requires a SUN JDK.
n
It is integrated in the file [path of application]/nl6/customer.sh
If you obtain a blank page:
Has the Neolane Web module started up? You should obtain:
nlserver pdump
11:00:39 >
Application server for Neolane Version 6.X (build XXXX) dated
DD/MM/YYYY
[...]
web@default (27515) - 55.2 Mb
[...]
If not, restart it using the following command:
Neolane v6.1 - Production Guide - Troubleshooting | 85
Neolane
nlserver start web
Note:
If you obtain a response of the following type when you list the Neolane modules:
nlserver pdump
11:00:39 >
DD/MM/YYYY
No tasks
Application server for Neolane Version 6.X (build XXXX) dated
You must restart the entire Neolane application. To do this, use the following commands:
n
If you have root or administrator access:
n
n
n
In Linux: /etc/init.d/nlserver6 start
In Windows: net start nlserver6
If not, then in the Neolane account:
nlserver watchdog -svc -noconsole
Connection thresholds
For heavily loaded servers, the connection threshold might be exceeded. In any event, it is useful to find out
why.
There are three different thresholds:
1 The Web connection threshold, configured in your web server. To modify it, contact your system
administrator.
2 The database connection threshold. To modify it, contact your database administrator.
3 The Neolane connection threshold, available in two places:
1 Tomcat side: all queries actually arriving on the Neolane Tomcat client.
This threshold is configured in the nl6/tomcat-7/conf/server.xml file. The maxProcessors
attribute lets you increase the threshold of the number of queries processed at a time. It can be
changed to 250, for instance.
<Connector className="org.apache.coyote.tomcat7.CoyoteConnector"
port="8080" minProcessors="5" maxProcessors="75"
enableLookups="true" redirectPort="8443"
acceptCount="100" debug="0" connectionTimeout="20000"
useURIValidationHack="false" disableUploadTimeout="true" />
<!-- Note : To disable connection timeouts, set connectionTimeout value to -1
-->
</Connector>
2 Database: set of all connections open at the same time on the database.
This threshold is configured in the file nl6/tomcat-4/conf/server.xml. The Pool attribute located
in dataSource lets you increase the threshold of queries processed simultaneously. For example, it
can be increased to 90:
<pool aliveTestDelaySec="600" freeCnx="0" maxCnx="90" maxIdleDelaySec="1200"/
Dr Watson in Windows XP (client)
The Dr Watson tool can block the re-starting of Neolane in Windows: you need to disable this
To do this:
1 Open a DOS command window via the following menu: Start>Run in Windows.
86 | © Neolane 2013
Troubleshooting
2 Enter the following command: start Drwtsn32
3 Disable notifications:
Stack trace in Linux
A stack trace represents a trace contained in a core type file. This file is generated in the event of a machine
error. It can identify the origin of the error.
Note:
n
n
A core file is named core.<NUM>.
gdb - The GNU Debugger must be installed on the machine.
Neolane technical support can ask you for this stack trace. To obtain it, enter the following commands in
Linux:
su - neolane
gdb nlserver <coreFile
//You get lots of output but the stack trace (or Back trace) can be obtained with :
(gdb) bt
// and forward all the output :
#0 0x0836c189 in ObjectValue::SetPropertyTarget ()
#1 0x082623b3 in CXtkScriptProperty::VDeclareProperties ()
#2 0x0826a835 in CXtkScriptContext::OnGetProperty ()
#3 0x08370b3d in JavaScriptGetProperty ()
#4 0x557b8aa7 in js_Interpret () from /usr/local/neolane/nl6/lib/libmozjs.so
#5 0x557afb97 in js_Execute () from /usr/local/neolane/nl6/lib/libmozjs.so
#6 0x5578921f in JS_ExecuteScript () from /usr/local/neolane/nl6/lib/libmozjs.so
#7 0x08370468 in JavaScriptSource::Evaluate ()
#8 0x0848fa03 in JSTDeliveryContext::ProcessScript ()
#9 0x0848fcb6 in JSTDeliveryContext::ProcessTemplate ()
#10 0x08460d2b in CDeliveryToolbox::CreateMailMessage ()
#11 0x080d51fe in CMtaQueue::PrepareMessages ()
#12 0x080d2b07 in CMtaQueue::Prepare ()
#13 0x080dca38 in CMtaChild::OnRun ()
#14 0x0814792c in ThreadStartRoutine ()
#15 0x55575b63 in start_thread () from /lib/tls/libpthread.so.0
#16 0x5565918a in clone () from /lib/tls/libc.so.6
Neolane technical support might ask you to run this command using a specific executable (to be supplied by
us).
Neolane v6.1 - Production Guide - Troubleshooting | 87
Neolane
In this case, simply run the following command:
gdb nlserver <coreFile>
by replacing nlserver with the executable provided by Neolane
For example:
gdb nlserver.1823 <coreFile>
Encoding of the Oracle database
Neolane uses the Oracle default encodings. If you use different ones, an incompatibility can occur. For more
on this, refer to the Installation guide.
Reactivating the console update request
If you selected the Do not request console update option and you wish to reactivate the update request,
apply the following procedure:
1 Open the editor of the registry database using the regedit command in the Windows Start>Execute
menu.
2 In the tree, display the options of the \HKEY_CURRENT_USER\Software\Neolane\NL_6\nlclient
node.
3 Delete the confAdvisedUpgrade entry and close the Registry editor.
88 | © Neolane 2013
Troubleshooting
Lost password
You can change or recover a lost password
There are two possible scenarios:
n
Password lost by a Neolane operator.
In this case, you can change the password of the operator concerned. To do this, connect via an operator
with administrator rights and replace the operator password.
n
Internal password loss.
If the internal password is lost, you must reinitialize it. To do this, apply the following procedure:
1 Edit the /usr/local/neolane/nl6/conf/serverConf.xml file.
2 Go to the internalPassword line.
<!-- XTK authentication mode internalPassword :
<xtk internalPassword="myPassword"/>
Password of internal account -->
3 Delete the string in quotes, in this case: myPassword
You thus obtain the following line:
!-- XTK authentication mode internalPassword :
<xtk internalPassword=""/
Password of internal account -->
4 Save changes and close the file.
5 Configure the new password. To do this, enter the following commands:
nlserver config -internalpassword
11:00:39 >
Application server for Neolane Version 6.X (build XXXX) of dd/mm/yyyy
Enter current password.
Password: (empty)
Enter the new password.
Password:
Confirmation
6 You can now use your new password to connect in Internal mode.
Abnormal JSP behavior
If certain jsp jobs are not successfully executed, you must force them to recompile.
For this, enter the following commands:
nlserver stop web
cd nl6/tomcat-7
rm -r work/
nlserver start web
Neolane v6.1 - Production Guide - Troubleshooting | 89
Neolane
The jsp jobs are regenerated the next time you connect.
90 | © Neolane 2013
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising