CloverETL 4.6.x Server Reference Manual

Configuration of CloverETL Server on Apache Tomcat (p. 14)

Installation of Apache Tomcat

See Application Server (p. 5) in system requirements for currently supported version of Apache Tomcat.

If you have Apache Tomcat already installed, you can skip this section.

In case of installation as Windows Service skip this section and continue with

Apache Tomcat as a Windows

Service (p. 12).

1. Download the binary distribution from http://tomcat.apache.org/download-60.cgi

(Tomcat 6) or http:// tomcat.apache.org/download-70.cgi

(Tomcat 7) or http://tomcat.apache.org/download-80.cgi

(Tomcat 8).

2. After you download the zip file, unpack it.

3. Run Tomcat by

[tomcat_home]/bin/startup.sh

(or

[tomcat_home]/bin/startup.bat

Windows OS).

4. Check whether Tomcat is running on URL: http://localhost:8080/ . Apache Tomcat info page should appear.

5. Apache Tomcat is installed.

If you need detailed installation instructions, go to: http://tomcat.apache.org/tomcat-6.0-doc/setup.html

(Tomcat

6) or to http://tomcat.apache.org/tomcat-7.0-doc/setup.html

(Tomcat 7) or to http://tomcat.apache.org/tomcat-8.0doc/setup.html

(Tomcat 8).

Continue with:

In case of installation on IBM AS/400 continue with Apache Tomcat on IBM AS/400 (iSeries) (p. 13).

Chapter 3. Installing

Apache Tomcat as a Windows Service

1. Download the binary distribution from http://tomcat.apache.org/download-60.cgi

(Tomcat 6) or http:// tomcat.apache.org/download-70.cgi

(Tomcat 7) or http://tomcat.apache.org/download-80.cgi

(Tomcat 8). Use the file called Windows Service Installer.

2. Use the standard installation wizard to install Apache Tomcat.

Important

The instructions mentioned bellow assume that Tomcat (as a service) is already installed. If this is not true, it is needed to do this first (using of Tomcat's Windows Service Installer is recommended).

Set up

JAVA_HOME

to point to the correct Java version: right click on This computer (or Computer on some Windows versions) and choose Properties

→

Advanced System Settings

→

Advanced

→

Environment

variables. The restart of operating system is needed to apply changes.

In case that Tomcat is installed as a Windows service, CloverETL configuration is performed using configuration of the respective service. The configuration can be performed either by graphical utility

[tomcat_home]/bin/

Tomcat8w.exe

or by command line utility

[tomcat_home]/bin/Tomcat8.exe

Graphical configuration utility

When using graphical configuration utility, select Java tab and set initial and maximum heap size in Initial

memory pool and Maximum memory pool fields. Other configuration parameters can be placed in Java Options field, being separated by new line. Then click on Apply and restart the service.

The Java tab allows you to use alternative Java virtual machine by setup of path to jvm.dll

file.

Command line tool

When using command line tool, navigate to [tomcat_home]/bin and stop the service (if running) using

(supposing that service is named "Tomcat8"):

.\Tomcat8.exe //SS//Tomcat8

Then configure service using command:

.\Tomcat8.exe //US//Tomcat8 --JvmMs=512 --JvmMx=1024

--JvmOptions=-Dclover.config.file=C:\path\to\clover-config.properties#-XX:MaxMetaspaceSize=256m

If you use Java 7, change -XX:MaxMetaspaceSize to -XX:MaxPermSize .

The parameter JvmMs is the initial and JvmMx is the maximum heap size in MB, JVM options are separated by

'#' or ';'. Finally start the service from Windows administration console or using:

.\Tomcat8.exe //TS//Tomcat8

When Apache Tomcat is run as a Windows service, it is not automatically available for tools like JConsole or

JVisualVM to attach and monitor Java process. However, those tools can still connect to the process via JMX. In order to expose Tomcat's Java process via JMX add the following options to the service settings:

-Dcom.sun.management.jmxremote.port=3333

-Dcom.sun.management.jmxremote.ssl=false

-Dcom.sun.management.jmxremote.authenticate=false

Once the service is run with those options you can connect to port 3333 using JMX and monitor server.

Chapter 3. Installing

Note

The same instructions can be used with Tomcat 6 and tomcat 7, just replace the number 8 with the number 6 or 7.

More information about run of Java applications as Windows Service can be found at http://commons.apache.org/ proper/commons-daemon/procrun.html

Continue with:

Apache Tomcat on IBM AS/400 (iSeries)

To run CloverETL Server on the iSeries platform, there are some additional settings:

1. Declare you are using Java 7.0 32-bit

2. Run Java with parameter

-Djava.awt.headless=true

To configure this you can modify/create a file

[tomcat_home]/bin/setenv.sh

which contains:

JAVA_HOME=/QOpenSys/QIBM/ProdData/JavaVM/jdk70/32bit

JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"

Continue with:

Installation of Apache Tomcat (p. 11) for details.

Installation of CloverETL Server

1. Download the web archive file ( clover.war

) containing CloverETL Server for Apache Tomcat.

2. Check if you meet prerequisites:

• Oracle JDK or JRE v. 1.7.x or higher. We recommend JDK 1.8.x.

•

JAVA_HOME

JRE_HOME

environment variable has to be set.

• Apache Tomcat 6.0.x or 7.0.x or 8.0.x is installed. CloverETL Server is developed and tested with the Apache

Tomcat 6.0.x, 7.0.x and 8.0.x containers (it may work unpredictably with other versions). See

• It is strongly recommended you change default limits for the heap

and perm gen

memory spaces.

See section

Configuration of CloverETL Server on Apache Tomcat (p. 14)

You can set the minimum and maximum memory heap size by adjusting the "Xms" and "Xmx" JVM parameters. You can set JVM parameters for Tomcat by setting the environment variable

JAVA_OPTS

in the

[TOMCAT_HOME]/bin/setenv.sh

file (if it does not exist, you may create it).

Create setenv file:

Unix-like systems: [tomcat]/bin/setenv.sh

export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxMetaspaceSize=512m -Xms128m -Xmx1024m" export CATALINA_OPTS="$CATALINA_OPTS -Dderby.system.home=$CATALINA_HOME/temp -server" echo "Using CATALINA_OPTS: $CATALINA_OPTS"

Windows systems:

[tomcat]/bin/setenv.bat

Chapter 3. Installing set "CATALINA_OPTS=%CATALINA_OPTS% -XX:MaxMetaspaceSize=512m -Xms128m -Xmx1024m" set "CATALINA_OPTS=%CATALINA_OPTS% -Dderby.system.home=%CATALINA_HOME%/temp -server" echo "Using CATALINA_OPTS: %CATALINA_OPTS%"

If you use Java 7, change

-XX:MaxMetaspaceSize

-XX:MaxPermSize

As visible in the settings above, there is also switch -server . For performance reasons, it is recommended to run the container in the "server" mode.

3. Copy clover.war

(which is built for Tomcat) to

[tomcat_home]/webapps

directory.

Please note, that copying is not an atomic operation. If Tomcat is running, mind duration of the copying process!

Too long copying might cause failure during deployment as Tomcat tries to deploy an incomplete file. Instead, manipulate the file when the Tomcat is not running.

4. War file should be detected and deployed automatically without restarting Tomcat.

5. Check whether CloverETL Server is running on URLs:

• Web-app root http://[host]:[port]/[contextPath]

The default Tomcat port for the http connector is 8080 and the default contextPath for CloverETL Server is "clover", thus the default URL is: http://localhost:8080/clover/

• Web GUI http://[host]:[port]/[contextPath]/gui

The default Tomcat port for the http connector is 8080 and the default contextPath for CloverETL Server is "clover", thus the default URL is: http://localhost:8080/clover/gui

Use default administrator credentials to access the web GUI: user name - "clover", password - "clover".

Continue with:

Configuration of CloverETL Server on Apache Tomcat

Default installation (without any configuration) is only recommended for evaluation purposes. For production use, both DB data source and SMTP server configurations are recommended.

The connection to database is configured in property file. On Apache Tomcat, it is

$CATALINA_HOME/ webapps/clover/WEB-INF/config.properties

file.

Properties File on Specified Location

Example of such a file: jdbc.driverClassName=...

jdbc.url=...

jdbc.username=...

jdbc.password=...

jdbc.dialect=...

license.file=/path/to/license.dat

Chapter 3. Installing

Properties file is loaded from location specified by a system property or by an environment variable clover_config_file ( clover.config.file

On Apache Tomcat, you can set the system property in the

[TOMCAT_HOME]/bin/setenv.sh

file (if it does not exist, you may create it). Just add:

JAVA_OPTS="$JAVA_OPTS -Dclover_config_file=/path/ to/cloverServer.properties"

If you need an example of connection to any of supported databases, see Chapter 8, Examples of DB Connection

Configuration (p. 53).

Continue with:

Installation of CloverETL Server (p. 15)

Jetty

Configuration of CloverETL Server on Jetty (p. 15)

Installation of CloverETL Server

1. Download the web archive file ( clover.war

) containing the CloverETL Server application which is built for Jetty.

2. Check if prerequisites are met:

• Oracle JDK or JRE (See Java Virtual Machine (p. 5) for required Java version.)

• See Application Server (p. 5) in system requirements for currently supported versions of Jetty.

All Jetty releases are available at http://download.eclipse.org/jetty/ .

• Memory allocation settings

It involves JVM parameters: -Xms -Xmx (heap memory) and -XX:MaxMetaspaceSize (classloaders memory limit). See section

You can set the parameters by adding

JAVA_OPTIONS='$JAVA_OPTIONS -Xms128m -Xmx1024m -XX:MaxMetaspaceSize=256m' to [jetty-home]/bin/jetty.sh

On Windows OS, edit [jetty-home]/start.ini

and add those parameters on the end of the file, each parameter on a new line.

If you use Java 7, change

-XX:MaxMetaspaceSize

-XX:MaxPermSize

3. Copy clover.war

[jetty-home]/webapps

4. Run

[jetty-home]/bin/jetty.sh start

(or java -jar start.jar --exec

on Windows

OS).

Finally, you can check if the server is running e.g. on http://localhost:8080/clover/ .

Configuration of CloverETL Server on Jetty

Default installation (without any configuration) is only recommended for evaluation purposes. For production use, both DB data source and SMTP server configurations are recommended.

Chapter 3. Installing

There are more ways to set config properties, yet the most common one is a properties file in a specified location.

Properties file in Specified Location

Example of such a file: jdbc.driverClassName=...

jdbc.url=...

jdbc.username=...

jdbc.password=...

jdbc.dialect=...

license.file=/path/to/license.dat

Note

JDBC Driver must be JDBC 4 compliant and stored in the

[jetty-home]/lib/ext

The common properties file is loaded from a location which is specified by the environment/system property clover_config_file or clover.config.file

. This is a recommended way of configuring Jetty.

On Unix-like OS, you can set system property in the

[jetty-home]/bin/jetty.sh

file. Add:

JAVA_OPTIONS="$JAVA_OPTIONS -Dclover_config_file=/path/to/cloverServer.properties"

On Windows OS add the property to

[jetty-home]/start.ini

, just after the memory settings.

Continue with:

Installation of CloverETL Server (p. 16)

IBM WebSphere

Configuration of CloverETL Server on IBM WebSphere (p. 18)

Installation of CloverETL Server

1. Get the web archive file ( clover.war

) with CloverETL Server application which is built for WebSphere.

2. Check if you meet prerequisites:

• IBM Java SDK (See

Java Virtual Machine (p. 5) for required Java version.)

Important

In order to ensure reliable function of CloverETL Server always use the latest version of IBM Java

SDK. At least SDK 7.0 SR6 (package IBM WebSphere SDK Java Technology Edition V7.0.6.1) is recommended. Using older SDKs may lead to deadlocks during execution of specific ETL graphs.

• See

Application Server (p. 5) in system requirements for currently supported version of IBM WebSphere.

(see http://www.ibm.com/developerworks/downloads/ws/was/ )

• Memory allocation settings

It involves JVM parameters: -Xms and -Xmx See section

You can set heap size in IBM WebSphere's Integrated Solutions Console (by default accessible at: http://[host]:9060/ibm/console/

)

Chapter 3. Installing

• Go to Servers →

Server Types

→

WebSphere application servers

→

[server1] (or another name of your server)

→

Java and Process Management

→

Process definition

→

Java Virtual Machine

• There is the Maximum heap size field. Default value is only 256 MB, which is not enough for ETL transformations.

Figure 3.1. Adjusting Maximum heap size limit

• On the same page, there is Generic JVM arguments. Add the perm space limit there, e.g. like this:

-XX:MaxPermSize=512M

Add the direct memory limit, e.g. like this:

-XX:MaxDirectMemorySize=512M

• Java runtime settings

Go to Servers

→

Application servers

→

[server1] (or another name of your server)

→

Java SDKs Here select version 1.7 as the default SDK.

• Save the changes to configuration and restart the server so that they take effect.

3. Deploy WAR file

• Go to Integrated Solutions Console

( http://localhost:9060/ibm/console/ )

• Go to Applications →

New Application

→

New Enterprise Application Here select a WAR archive of the

CloverETL server and deploy it to the application server, but do not start it.

4. Configure application class loading

Chapter 3. Installing

Go to WebSphere Enterprise Applications

→

clover_war (or other name of the Clover application)

→

Manage Modules

→

CloverETL Here under Class loader order select Classes loaded with local class

loader first (parent last).

5. Save the changes to the server configuration and start the clover_war application.

6. Check whether the server is running

Provided you set clover.war

as the application running with "clover" context path. Notice the port number has changed: http://localhost:9080/clover

Note

Please note, that some CloverETL features using third party libraries don't work properly on IBM

WebSphere

• Hadoop is guaranteed to run only on Oracle Java 1.6+, but Hadoop developers do make an effort to remove any Oracle/Sun-specific code. See Hadoop Java Versions on Hadoop Wiki.

• AddressDoctor5 on IBM WebSphere, requires additional JVM parameter

-Xmso2048k

to prevent AddressDoctor from crashing JVM. See documentation on AddressDoctor component.

Configuration of CloverETL Server on IBM WebSphere

Default installation (without any configuration) is only recommended for evaluation purposes. For production use, configuring at least the DB data source and SMTP server is recommended.

There are more ways to set config properties. The most common one is a properties file in the specified location.

Properties File in Specified Location

Example of such a file: jdbc.driverClassName=...

jdbc.url=...

jdbc.username=...

jdbc.password=...

jdbc.dialect=...

license.file=/path/to/license.dat

Set system property (or environment variable) clover_config_file

pointing to the config properties file.

• go to Integrated Solutions Console

( http://localhost:9060/ibm/console/ )

• Go to Servers →

WebSphere application servers

→

[server-name]

→

Java and Process Management

→

Process Definition

→

Java Virtual Machine

→

Custom Properties

• Create system property named clover_config_file

whose value is full path to config file named e.g.

cloverServer.properties

on your file system.

• This change requires restarting IBM WebSphere.

Continue with:

Installation of CloverETL Server (p. 19)

Chapter 3. Installing

Glassfish / Sun Java System Application Server

Configuration of CloverETL Server on Glassfish (p. 19)

Important

Glassfish 3.1.2 contains a bug causing Launch Services to work improperly (see https://java.net/jira/ browse/GLASSFISH-18444 ). Use a different version of Glassfish (version 3.1.2.2 is recommended).

Installation of CloverETL Server

1. Get CloverETL Server web archive file ( clover.war

) which is built for Glassfish 3.

2. Check if you meet prerequisites

• Oracle JDK or JRE (See Java Virtual Machine (p. 5) for required java version.)

• Glassfish (CloverETL Server is tested with version 3.1.2.2)

• Memory allocation settings

It involves JVM parameters: -Xms -Xmx and -XX:MaxPermSize See section

Memory

Settings (p. 38) for details.

You can set the heap size and perm space in XML file

[glassfish-home]/glassfish/domains/ domain1/config/domain.xml

Add or change these sub-elements to

<java-config>

<jvm-options>-XX:MaxPermSize=384m</jvm-options>

<jvm-options>-XX:PermSize=256m</jvm-options>

<jvm-options>-Xms512m</jvm-options>

<jvm-options>-Xmx2g</jvm-options>

These changes require restarting Glassfish to take effect.

3. Deploy WAR file

• Open Glassfish Administration Console

It is accessible at http://localhost:4848/ by default.

• Go to Applications →

Web Applications and click Deploy ....

• Upload WAR file with CloverETL server application or select the file from filesystem if it is present on the machine running Glassfish.

• Fill in attributes Application name and Context Root with "clover".

• Submit form.

Configuration of CloverETL Server on Glassfish

Default installation (without any configuration) is only recommended for evaluation purposes. For production use, configuring at least the DB data source and SMTP server is recommended.

There are more ways to set config properties. The most common one is a properties file in the specified location.

Chapter 3. Installing

Properties File in Specified Location

Example of such a file: datasource.type=JDBC jdbc.driverClassName=org.postgresql.Driver

jdbc.url=jdbc:postgresql://localhost:5432/cloverdb jdbc.username=root jdbc.password= jdbc.dialect=org.hibernate.dialect.PostgreSQLDialect

license.file=/path/to/license.dat

Set system property clover.config.file

pointing to the config properties file:

• Go to Glassfish Administration Console

By default accessible at http://localhost:4848/

• Go to Configuration →

System Properties

• Create system property named clover.config.file

whose value is a full path to the file on your file system named e.g.

/home/clover/cloverServer.properties

Copy the JDBC driver JAR file for a selected database (PostgreSQL) into

[glassfish]/glassfish/ domains/[domain-name]/lib

These changes require restarting Glassfish to take effect.

Continue with:

Installation of CloverETL Server (p. 20)

JBoss Application Server

Configuration of CloverETL Server on JBoss AS (p. 21)

Installation of CloverETL Server

1. Get CloverETL Server web archive file ( clover.war

) that is built for JBoss AS.

2. Check if you meet prerequisites

• Oracle JDK or JRE (See Java Virtual Machine (p. 5) for required Java version.)

• JBoss AS 6.x http://www.jboss.org/jbossas/downloads

• Memory settings for JBoss Java process. See section

5. Configure CloverETL Server according to the description in the next section (p. 21) .

You can set the memory limits in

[jboss-home]/bin/run.conf

( run.conf.bat

on Windows

OS)

JAVA_OPTS="$JAVA_OPTS -XX:MaxMetaspaceSize=512m -Xms128m -Xmx2048m"

If you use Java 7, change

-XX:MaxMetaspaceSize

-XX:MaxPermSize

On Windows, perform steps analogic to the ones above.

Chapter 3. Installing

3. Create a separate JBoss server configuration

However it may be useful to use a specific JBoss server configuration, when it is necessary to run CloverETL:

• isolated from other JBoss applications

• with a different set of services

• with different libraries on the classpath than other applications

See the JBoss manual for details about the JBoss server configuration: JBoss Server Configurations Start the Server With Alternate Configuration

4. Configure database connection

As CloverETL Server's embedded Derby database does not work under JBoss AS, a database connection has to be always configured. We used MySQL accessed via JNDI-bound datasource in this example:

• Create datasource deployment file

[jboss-home]/server/[serverConfiguration]/ deploy/mysql-ds.xml

<local-tx-datasource>

<jndi-name>CloverETLServerDS</jndi-name>

<connection-url>jdbc:mysql://localhost:3306/cloverServerDB</connection-url>

<driver-class>com.mysql.jdbc.Driver</driver-class>

<user-name>root</user-name>

</local-tx-datasource>

</datasources>

Note

Special characters in the XML file have to be typed in as XML entities. For instance, ampersand

"&" as "&" etc.

"CloverETLServerDS" is the name under which the datasource will be accessible. The thing to do here is to set database connection parameters ( connection-url

, driver-class

, user-name

and password

) to the database. The database has to be empty before the first execution, the server creates its tables on its own.

• Put the JDBC driver JAR file for your database to the application server classpath. In this example we copied the file mysql-connector-java-5.1.5-bin.jar

[jboss-home]/server/

[serverConfiguration]/lib

6. Deploy the WAR file

Copy clover.war

[jboss-home]/server/[serverConfiguration]/deploy

7. Start JBoss AS via

[jboss-home]/bin/run.sh

(or run.bat

on Windows OS) If you want to run JBoss with a specific server configuration, it has to be specified as a parameter:

[jboss-home]/bin/run.sh c [serverConfiguration]

If the serverConfiguration isn't specified, the "default" is used.

Configuration of CloverETL Server on JBoss AS

Default installation (without any configuration) does not work under JBoss AS. In order to be able to use the

CloverETL Server, a working database connection is required.

Chapter 3. Installing

There are more ways to set configuration properties. The most common one is a properties file in a specified location.

Properties File in Specified Location

• Create cloverServer.properties

in a suitable (i.e. readable by JBoss AS process) directory datasource.type=JNDI datasource.jndiName=java:/CloverETLServerDS jdbc.dialect=org.hibernate.dialect.MySQLDialect

license.file=/home/clover/config/license.dat datasource.type

indicates the server will use JNDI-bound datasource created in steps above. The property datasource.jndiName

specifies where is the datasource to be found in JNDI. Set JDBC dialect according

to your database server ( Part III, “ Configuration ” (p. 42) ). You can set path to your license file, too.

• Set system property (or environment variable) clover.config.file

It should contain the full path to the cloverServer.properties

file created in the previous step.

The simplest way is setting Java parameter in [jboss-home]/bin/run.sh

, e.g.: export JAVA_OPTS="$JAVA_OPTS -Dclover.config.file=/home/clover/config/cloverServer.properties"

Do not override other settings in the

JAVA_OPTS

property - i.e. memory settings described above.

On Windows OS, edit

[jboss-home]/bin/run.conf.bat

and add this line to the section where options are passed to the JVM: set JAVA_OPTS=%JAVA_OPTS% -Dclover.config.file=C:/JBoss6/cloverServer.properties

• Restart the JBoss AS so that the changes take effect.

• Check the CloverETL Server application is running:

Server's console is accessible at http://localhost:8080/clover by default.

Continue with:

Installation of CloverETL Server (p. 22)

JBoss Enterprise Application Platform

Configuration of CloverETL Server on JBoss EAP (p. 24)

Installation of CloverETL Server

1. Get CloverETL Server web archive file ( clover.war

) which is built for JBoss EAP.

2. Check if you meet prerequisites

• Oracle JDK or JRE (See Java Virtual Machine (p. 5) for required Java version.)

• JBoss EAP 6.x (JBoss AS 7) - see http://www.jboss.org/jbossas/downloads

Chapter 3. Installing

• Memory settings for JBoss Java process. See section

6. Configure CloverETL Server according to a description in the next section (p. 24) .

You can set the memory limits in

[jboss-home]/bin/standalone.conf

(

[jboss-home]/ bin/standalone.conf.bar

on Windows OS) for JBoss EAP in standalone mode:

JAVA_OPTS="$JAVA_OPTS -XX:MaxMetaspaceSize=512m -Xms128m -Xmx2048m"

If you use Java 7, change -XX:MaxMetaspaceSize to -XX:MaxPermSize .

On Windows, perform steps analogic to the ones above.

3. Configure database connection

By default CloverETL Server uses embedded Derby database, however such setup is not recommended for production use. You can use database connection provided by JNDI-bound datasource deployed by JBoss EAP. In order to define the datasource, edit the file

[jbosshome]/standalone/configuration/standalone.xml

and into section

under element

add definition of the datasource. Here is an example of datasource connecting to MySQL database:

<datasource jndi-name="java:jboss/datasources/CloverETLServerDS"

pool-name="CloverETLServerDS-Pool" enabled="true">

<connection-url>jdbc:mysql://localhost:3307/cloverServerDB</connection-url>

<driver>com.mysql</driver>

<transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>

<pool>

<min-pool-size>5</min-pool-size>

<max-pool-size>50</max-pool-size>

</pool>

<user-name>root</user-name>

</security>

<prepared-statement-cache-size>32</prepared-statement-cache-size>

<share-prepared-statements>true</share-prepared-statements>

</statement>

</datasource>

<driver-class>com.mysql.jdbc.Driver</driver-class>

</driver>

</drivers>

4. The datasource definition references a module ( mysql.driver

) with MySQL JDBC driver. Take following steps to add the module:

5. Under JBoss EAP there are more options to setup CloverETL Server's database: along with JNDI-bound data source it is possible to use embedded Derby database or other supported database specified in CloverETL configuration file.

In order to be able to connect to the database, one needs to define global module so that the driver is available for CloverETL web application - copying the driver to the lib/ext

directory of the server will not work. Such module is created and deployed in few steps (the example is for MySQL and module's name is mysql.driver

• Create directory

[jboss-home]/modules/mysql/driver/main

(note that the directory path corresponds to module name mysql.driver

)

Chapter 3. Installing

• Copy the driver mysql-connector-java-5.1.5-bin.jar

to that directory and create there file module.xml

with following content:

<?xml version="1.0" encoding="UTF-8"?>

<resource-root path="mysql-connector-java-5.1.5-bin.jar" />

</resources>

</dependencies>

</module>

• Add the module to global server modules: in case of the standalone JBoss EAP server they are defined in

[jboss-home]/standalone/configuration/standalone.xml

. The module is to be added into EE domain subsystem section:

<global-modules>

</global-modules>

<spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>

<jboss-descriptor-property-replacement>true</jboss-descriptor-property-replacement>

</subsystem>

7. Deploy WAR file

Copy the file clover.war

[jboss-home]/standalone/deployments

8. Run

[jboss-home]/bin/standalone.sh

(or standalone.bat

on Windows OS) to start the JBoss platform.

It may take a couple of minutes until all applications are started.

9. Check JBoss response and CloverETL Server response

• JBoss administration console is accessible at http://localhost:8080/ by default. Default username/password is admin/admin

• CloverETL Server is accessible at http://localhost:8080/clover by default.

Configuration of CloverETL Server on JBoss EAP

Default installation (without any configuration) is only recommended for evaluation purposes. For production use, configuring at least the database connection and SMTP server connection is recommended.

There are more ways to set config properties. The most common one is a properties file in a specified location.

Properties File in Specified Location

• Create cloverServer.properties

in a directory the JBoss EAP has right to read:

Chapter 3. Installing datasource.type=JNDI datasource.jndiName=java:jboss/datasources/CloverETLServerDS jdbc.dialect=org.hibernate.dialect.MySQLDialect

license.file=/home/clover/config/license.dat

Do not forget to set correct JDBC dialect according to your database server ( Part III, “ Configuration ” (p. 42)

). You can set the path to the license file, too.

• Alternatively, you can set "JDBC" datasource.type

and configure the database connection to be managed directly by CloverETL Server (provided that you have deployed proper JDBC driver module to the server): datasource.type=JDBC jdbc.url=jdbc:mysql://localhost:3306/cloverServerDB jdbc.dialect=org.hibernate.dialect.MySQLDialect

jdbc.driverClassName=com.mysql.jdbc.Driver

jdbc.username=root jdbc.password=root license.file=/home/clover/config/license.dat

• Set system property (or environment variable) clover.config.file

It should contain the full path to the cloverServer.properties

file created in the previous step.

The simplest way to set the system property is to edit the configuration file

[jboss-home]/standalone/ configuration/standalone.xml

, and to add the following snippet just under

<extensions> section:

<system-properties>

</system-properties>

• Restart the JBoss EAP so that the changes take effect.

• Check the CloverETL Server application is running:

Server's console is accessible at http://localhost:8080/clover by default.

Note

The JBoss EAP has, by default, enabled HTTP session replication. This requires session serialization that is not supported by CloverETL Server, and produces lots of harmless errors in JBoss's console like this:

10:56:38,248 ERROR [org.infinispan.transaction.TransactionCoordinator] (http-/127.0.0.1:8080-2)

ISPN000188: Error while processing a commit in a two-phase transaction: java.lang.UnsupportedOperationException: Serialization of HTTP session objects is not supported by CloverETL Server - disable the session passivation/replication for this web application.

at com.cloveretl.server.web.gui.e.writeExternal(Unknown Source) [cs.jar:]

at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:874)

To get rid of these errors, disable the session replication. Edit

[jboss-home]/standalone/ configuration/standalone.xml

and comment out whole

<cache-container name="web" aliases="standard-session-cache">

block under

section.

Chapter 3. Installing

Continue with:

Installation of CloverETL Server (p. 26)

Oracle WebLogic Server

Configuration of CloverETL Server on WebLogic (p. 27)

Installation of CloverETL Server

1. Get CloverETL Server web archive file ( clover.war

) which is built for WebLogic.

2. Check if you meet prerequisites

• Oracle JDK or JRE (See Java Virtual Machine (p. 5) for required Java version.)

• WebLogic (CloverETL Server is tested with WebLogic Server 11g (10.3.6) and WebLogic Server 12c

(12.1.2), see http://www.oracle.com/technetwork/middleware/ias/downloads/wls-main-097127.html)

WebLogic has to be running and a domain has to be configured. You can check it by connecting to

Administration Console: http://hostname:7001/console/ (7001 is the default port for HTTP). Username and password are specified during installation.

• Memory allocation settings

It involves JVM parameters: -Xms -Xmx and -XX:MaxMetaspaceSize

See section

Configuration of CloverETL Server on

You can set it i.e. with export JAVA_OPTIONS='$JAVA_OPTIONS -Xms512m -Xmx2048m -XX:MaxMetaspaceSize=512m' to the start script

If you use Java 7, change

-XX:MaxMetaspaceSize

-XX:MaxPermSize

This change requires restarting the domain.

3. Change HTTP Basic Authentication configuration

• When WebLogic finds "Authentication" header in an HTTP request, it tries to find a user in its own realm.

This behavior has to be disabled so CloverETL could authenticate users itself.

• Modify config file

[domainHome]/config/config.xml

. Add

<enforce-valid-basicauth-credentials>false</enforce-valid-basic-auth-credentials>

element into

<security-configuration>

element (just before the end tag).

4. Deploy a WAR file (or an application directory)

• Get a clover.war suitable for your WebLogic version.

• Deploy the clover.war using the WebLogic Server Administration Console. See the Oracle Fusion

Middleware Administrator's Guide (http://docs.oracle.com/cd/E23943_01/core.1111/e10105/toc.htm) for details.

5. Configure a license and other configuration properties. (See

WebLogic (p. 27))

6. Check a CloverETL Server URL

• Web-app is started automatically after deployment, so you can check whether it is up and running.

Chapter 3. Installing

CloverETL Server is accessible at http://host:7001/clover by default. The port 7001 is the default WebLogic

HTTP Connector port.

Configuration of CloverETL Server on WebLogic

Default installation (without any configuration) is only recommended for evaluation purposes. For production use, at least the DB data source and SMTP server configuration is recommended.

There are more ways to set config properties. The most common one is a properties file in a specified location.

Properties File in Specified Location

Create cloverServer.properties

in a suitable directory.

Config file should contain DB datasource config, SMTP connection config, etc.

Set clover_config_file

system property (or environment variable) pointing to the config properties file.

• Set JAVA_OPTIONS variable in the WebLogic domain start script

[domainHome]/startWebLogic.sh

JAVA_OPTIONS="${JAVA_OPTIONS} -Dclover_config_file=/path/to/clover-config.properties

• This change requires restarting WebLogic.

Important

When CloverETL Server is deployed on WebLogic and JNDI Datasource pointing to Oracle DB is used, there must be an extra config property in the config file: quartz.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.weblogic.WebLogicOracleDelegate

Continue with:

CloverETL Server License using a Web Form (p. 28).

Chapter 3. Installing

Installation of CloverETL Server License

To be able to execute graphs, CloverETL Server requires a valid license. You can install CloverETL Server without any license, but no graph will be executed.

There are three ways of installing the license working on all application servers.

• The simplest way is using 'Add license form' and save the license content to a database. See

Installation of

• The second way is configuring a license.file

property to set the plain license file. See

Installation of

CloverETL Server License using a license.file Property (p. 31).

• The last way is a separate web application clover-license.war

. See Separate License WAR (p. 31).

Note

CloverETL license can be changed any time by replacing license.dat

file. Afterwards, you have to let CloverETL Server know the license has changed.

• Go to server web GUI →

Configuration

→

Setup

→

License

• Click Reload license.

• Alternatively, you can restart the CloverETL Server application.

Note

All three ways listed below can be used at the same time. The most recent valid license is used.

Installation of CloverETL Server License using a Web Form

If the CloverETL Server has been started without assigning any license, you can use Add license form in the server gui to install it. In this case the hyperlink No license available in system. Add new license is displayed on the login page. It links to the form.

Chapter 3. Installing

Figure 3.2. Login page of CloverETL Server without license

You can paste a license text into License key or use Browse button to search for license file in the filesystem.

After clicking Update button the license is validated and saved to the database table clover_licenses. If the license is valid, a table with license's description appears. To proceed to CloverETL Serve console click Continue to

server console.

To skip adding a license you can use Close button.

Chapter 3. Installing

Figure 3.3. Add new license form

Update of CloverETL Server License in the Configuration Section

If the license has been already installed, you can still change it by using form in the server web gui.

• Go to server web GUI →

Configuration

→

CloverETL Info

→

License

• Click Update license.

You can paste a license text into a License key textarea or use the Browse button to search for a license file in the filesystem. To skip adding a license you can use Close button.

After clicking the Update button the license is saved to the database table clover_licenses and reloaded.

Chapter 3. Installing

Figure 3.4. Update license form

Note

The license in the database is common for all nodes in the cluster. Reloading of the license occurs on each node in the cluster.

Note

If you assign more valid licenses, the most recent one is used.

Installation of CloverETL Server License using a license.file Property

1. Get the license.dat

file.

2. Set the CloverETL Server license.file

parameter to the path to license.dat

. Set its value to full path to the license.dat

file.

See Chapter 9, List of Properties (p. 70) for list of properties.

3. Restart the application server.

Separate License WAR

Simple approach, but it may be used only for standalone server running on Apache Tomcat.

1. Download the clover-license.war

web archive file.

2. Copy clover-license.war

to the

[tomcat_home]/webapps

directory.

3. The war file should be detected and deployed automatically without restarting Tomcat.

4. Check whether the license web-app is running on:

Chapter 3. Installing http://[host]:[port]/clover-license/

(Note: clover-license

contextPath is mandatory and cannot by changed)

IBM InfoSphere MDM Plugin Installation

Downloading

IBM InfoSphere MDM Components for CloverETL Server are downloaded as a ZIP file containing the extension. The ZIP file is available for download under your account on www.cloveretl.com in CloverETL Server download area, under the Utilities section as ibm-mdm-connectors.${version}.zip

file.

Requirements

Requirements of IBM InfoSphere MDM Components:

• supported OS are Microsoft Windows 32 bit, Microsoft Windows 64 bit, Linux 64 bit, and Mac OS X Cocoa

• at least 512MB of RAM

• installed CloverETL Server

The support for 32 bit Linux was removed in 4.5.0.

Installation into Server

The following steps are needed to install IBM InfoSphere MDM Components into CloverETL Server:

1. Install CloverETL Server, see its documentation for details.

2. Download the ZIP file with IBM InfoSphere MDM Components for Server and store it on the system where

CloverETL Server is installed. See

Downloading (p. 32) for instructions for the download.

3. The ZIP file contains a CloverETL plugin. Your Server installation needs to be configured to find and load the plugin from the ZIP file. This is done by setting the engine.plugins.additional.src

Server configuration property to the absolute path of the ZIP file, e.g. engine.plugins.additional.src=c:/Server/ibm-mdm-connectors.4.6.0.zip

(in case the Server is configured via a property file).

Details for setting the configuration property depend on your Server installation specifics, application server used etc. See CloverETL Server documentation for details. Typically the property would be set similarly to how you set-up the properties for connection to the Server's database. Updating the configuration property usually requires restart of the Server.

4. To verify that the plugin was loaded successfully, login to the Server's Reporting Console and look in the Configuration > CloverETL Info > Plugins page. In the list of plugins you should see cloveretl.engine.initiate

Troubleshooting

If you get an Unknown component or Unknown connection error when running a graph with IBM

InfoSphere MDM components, it means that the IBM InfoSphere MDM Components plugin was not loaded by the Server successfully. Please check the above steps to install the plugin, especially the path to the ZIP file.

Chapter 3. Installing

Possible Issues during Installation

Since CloverETL Server is considered a universal JEE application running on various application servers, databases and jvm implementations, problems may occur during the installation. These can be solved with a proper configuration of the server environment. This section contains tips for the configuration.

Memory Issues on Derby (p. 33)

JAVA_HOME or JRE_HOME Environment Variables Are Not Defined (p. 33)

Apache Tomcat Context Parameters Do Not Have Any Effect (p. 34)

Tomcat Log File catalina.out Is Missing on Windows (p. 34)

Timeouts Waiting for JVM (p. 34)

clover.war as Default Context on WebSphere (Windows OS) (p. 35)

Tomcat 6.0 on Linux - Default DB (p. 35)

Derby.system.home Cannot be Accessed (p. 35)

Environment Variables and More than one CloverETL Server Instances Running on Single Machine (p. 36)

Special Characters and Slashes in Path (p. 36)

File System Permissions (p. 36)

JMS API and JMS Third-Party Libraries (p. 36)

Using an Unsupported JDBC Connector for MySQL (p. 36)

Memory Issues on Derby

If your server suddenly starts consuming too much resources (CPU, memory) despite having been working well before, it might be because of running the internal Derby DB. Typically, causes are incorrect/incomplete shutdown of Apache Tomcat and parallel (re)start of Apache Tomcat.

Solution: move to a standard (standalone) database.

How to fix this? Redeploy CloverETL Server:

1. Stop Apache Tomcat and verify there are no other instances running. If so, kill them.

2. Backup the config file, if you configured any.

3. Delete the webapps/clover

directory.

4. Start Apache Tomcat server. It will automatically redeploy Clover Server.

5. Verify you can connect from Designer and from web.

6. Shutdown Apache Tomcat.

7. Restore the config file and point it to your regular database.

8. Start Apache Tomcat.

JAVA_HOME or JRE_HOME Environment Variables Are Not Defined

If you are getting this error message during an attempt to start your application server (mostly Tomcat), perform the following actions.

Linux:

These two commands will help you set paths to the variables on the server.

• [root@server /] export JAVA_HOME=/usr/local/java

•

[root@server /] export JRE_HOME=/usr/local/jdk

As a final step, restart the application server.

Chapter 3. Installing

Windows OS:

Set

JAVA_HOME

to your JDK installation directory, e.g.

C:\Program Files\java\jdk1.8.0

. Optionally, set also

JRE_HOME

to the JRE base directory, e.g.

C:\Program Files\java\jre8

Important

If you only have JRE installed, specify only

JRE_HOME

Apache Tomcat Context Parameters Do Not Have Any Effect

Tomcat may sometimes ignore some of context parameters. It may cause weird CloverETL Server behaviour, since it looks like configured, but only partially. Some parameters are accepted, some are ignored. Usually it works fine, however it may occur in some environments. Such behaviour is consistent, so after restart it's the same. It's probably related to Tomcat issues: https://issues.apache.org/bugzilla/show_bug.cgi?id=47516 and https://issues.apache.org/bugzilla/show_bug.cgi?id=50700 To avoid this, please use a properties file instead of context parameters to configure CloverETL Server.

Tomcat Log File catalina.out Is Missing on Windows

Tomcat start batch files for Windows aren't configured to create catalina.out file which contains standard output of the application. The catalina.out

file may be vital when the Tomcat isn't started in console and any issue occurs. Or even when Tomcat is executed in the console, it may be closed automatically just after the error message appears in it.

Please follow these steps to enable catalina.out creation:

• Modify [tomcat_home]/bin/catalina.bat. Add parameter "/B" to lines where "_EXECJAVA" variable is set.

There should be two these lines. So they will look like this: set _EXECJAVA=start /B [the rest of the line]

Parameter /B causes, that "start" command doesn't open new console window, but runs the command it's own console window.

• Create new startup file. e.g. [tomcat_home]/bin/startupLog.bat, containing only one line: catalina.bat start > ..\logs\catalina.out 2<&1

It executes Tomcat in the usual way, but standard output isn't put to the console, but to the catalina.out file.

Then use the new startup file instead of

[tomcat_home]/bin/startup.bat

Timeouts Waiting for JVM

If you get the Jetty application server successfully running but cannot start Clover Server, it might be because of the wrapper waiting for JVM too long (it is considered a low-memory issue). Examine

[JETTY_HOME]\logs

\jetty-service.log

for a line like this:

Startup failed: Timed out waiting for signal from JVM.

If it is there, edit

[JETTY_HOME]\bin\jetty-service.conf

and add these lines: wrapper.startup.timeout=60 wrapper.shutdown.timeout=60

Chapter 3. Installing

If that does not help either, try setting 120 for both values. Default timeouts are 30 both.

clover.war as Default Context on WebSphere (Windows OS)

If you are deploying clover.war

on the IBM WebSphere server without context path specified, be sure to check whether it is the only application running in the context root. If you cannot start Clover Server on WebSphere, check the log and look for a message like this:

com.ibm.ws.webcontainer.exception.WebAppNotLoadedException:

Failed to load webapp: Failed to load webapp: Context root /* is already bound.

Cannot start application CloverETL

If you can see it, then this is the case. Getting rid of the issue, the easiest way is to stop all other (sample) applications and leave only clover.war

running on the server. That should guarantee the server will be available in the context root from now on (e.g. http://localhost:9080/).

Figure 3.5. Clover Server as the only running application on IBM WebSphere

Tomcat 6.0 on Linux - Default DB

When using the internal (default) database on Linux, your Clover Server might fail on first start for no obvious reasons. Chances are that the

/var/lib/tomcat6/databases

directory was not created (because of access rights in parent folders).

Solution: Create the directory yourself and try restarting the server. This simple fix was successfully tested with

Clover Server deployed as a WAR file via Tomcat web admin.

Derby.system.home Cannot be Accessed

If the server cannot start and the following message is in the log: java.sql.SQLException: Failed to start database 'databases/cloverserver'

Chapter 3. Installing then see the next exception for details. After that check settings of the derby.system.home

system property.

It may point to an unaccessible directory, or files may be locked by another process. We suggest you set a specific directory as the system property.

Environment Variables and More than one CloverETL Server Instances

Running on Single Machine

If you are setting environment variables like clover_license_file

or clover_config_file

, remember you should not be running more than one CloverETL Server. Therefore if you ever needed to run more

instances at once, use other ways of setting parameters (see Part III, “ Configuration ” (p. 42) for description of

all possibilities) The reason is the environment variable is shared by all applications in use causing them to share configurations and fail unexpectedly. Instead of environment variables you can use system properties (passed to the application container process using parameter with -D prefix: -Dclover_config_file).

Special Characters and Slashes in Path

When working with servers, you ought to stick to folder naming rules more than ever. Do not use any special characters in the server path, e.g. spaces, accents, diacritics are all not recommended. It's unfortunately common naming strategy on Windows systems. It can produce issues which are hard to find. If you are experiencing weird errors and cannot trace the source of them, why not install your application server in a safe destination like:

C:\JBoss6\

Similarly, use slashes but never backslashes in paths inside the

*.properties

files, e.g. when pointing to the

Clover Server license file. If you incorrectly use backlash, it will be considered an escape character and the server may not work fine. This is an example of a correct path: license.file=C:/CoverETL/Server/license.dat

File System Permissions

Application server must be executed by OS user which has proper read/write permissions on file system. Problem may occur, if app-server is executed by root user for the first time, so log and other temp files are created by root user. When the same app-server is executed by another user, it will fail because it cannot write to root's files.

JMS API and JMS Third-Party Libraries

Missing JMS libraries do not cause fail of server startup, but it is issue of deployment on an application server, thus it still suits to this chapter.

clover.war itself does not contain jms.jar, thus it has to be on an application server's classpath. Most of the application servers have jms.jar by default, but e.g. Tomcat does not. So if the JMS features are needed, the jms.jar

has to be added explicitly.

If "JMS Task" feature is used, there must be third-party libraries on a server's classpath as well. The same approach is recommended for JMS Reader/Writer components, even if these components allow to specify external libraries.

It is due to common memory leak in these libraries which causes "OutOfMemoryError: PermGen space".

Using an Unsupported JDBC Connector for MySQL

CloverETL Server requires MySQL 5 up to version 5.5 included. Using an unsupported JDBC connector for

MySQL might cause exception, for example:

Chapter 3. Installing could not execute query

You have an error in your SQL syntax; check the manual that coresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1

Chapter 4. Postinstallation Configuration

Memory Settings

Current implementation of Java Virtual Machine allows only global configuration of memory for the JVM system process. Thus whole application server, together with WARs and EARs running on it, share one memory space.

Default JVM memory settings is too low for running application container with CloverETL Server. Some application servers, like IBM WebSphere, increase JVM defaults themselves, however they still may be too low.

The best memory limits depend on many conditions, i.e. transformations which CloverETL should execute. Please note, that maximum limit isn't amount of permanently allocated memory, but limit which can't be exceeded. If the limit was exhausted, the OutOfMemoryError would be raised.

JVM Memory Areas

JVM memory consists of several areas: Heap memory, PermGen space (for loaded classes), so called direct memory (used by graph edges), and stack memory (allocated for each thread). JVM memory is not just HEAP memory. Thus it's not wise to set HEAP limit too high, since if it consumed whole RAM, JVM wouldn't be able to allocate direct memory and stack for new threads.

Configuring Memory

You can set the minimum and maximum memory heap size by adjusting the "Xms" and "Xmx" JVM parameters.

There are more ways to change the settings depending on the used application container.

Recommended Heap Memory Configuration

If you have no idea about the memory required for the transformations, a maximum of 1-2 GB heap memory is recommended. This limit may be increased during transformations development when

OutOfMemoryError occurs.

Heap limit is not limit of memory used by JVM. If you do not understand JVM in details, you should not assign more than 50% of main memory to heap.

Memory Configuration in Different Java Versions

In Java 7 and earlier, memory space for loading classes (so called "PermGen space") is separated from heap memory, and can be set by the JVM parameter "-XX:MaxPermSize". By default, it is just 64 MB which is not enough for enterprise applications. Again, suitable memory limit depends on various criteria, but 512 MB should be enough in most cases. If the PermGen space maximum is too low,

OutOfMemoryError: PermGen space may occur.

In Java 8, memory space for loading classes (so called "Metaspace") is separated from heap, and can be set by the

JVM parameter -XX:MaxMetaspaceSize. The default maximum Metaspace size is unlimited. You can change it by parameter

-XX:MaxMetaspaceSize

Please see the specific container section for details how to make the settings.

Codecache Size

Some clover server installations can occasionally run into performance issue: JVM is running more than hundred times slower. The issue can be caused by a full codecache (https://docs.oracle.com/javase/8/embedded/developapps-platforms/codecache.htm). Reserved code cache size is platform dependent and can be too small for clover server. It is highly recommended to increase code cache size using following JVM argument:

-XX:ReservedCodeCacheSize=256m

Chapter 4. Postinstallation

Configuration

Maximum Number of Open Files

When using resource-greedy components, such as FastSort, or when running a large number of graphs concurrently, you may hit the system limit on simultaneously open files. This is usually indicated by the java.io.IOException: Too many open files

exception.

The default limit is fairly low in many Linux distributions (e.g. 4096 in Ubuntu). Such a limit can be easily exceeded, considering that one FastSort component can open up to 1,000 files when sorting 10 million records.

Furthermore, some application containers recommend increasing the limit themselves (8192 for IBM WebSphere).

Therefore it is recommended to increase the limit for production systems. Reasonable limits vary from 10,000 to about 100,000 depending on the expected load of CloverETL Server and the complexity of your graphs.

The current limit can be displayed in most UNIX-like systems using the ulimit -Hn command.

The exact way of increasing the limit is OS-specific and is beyond the scope of this manual.

Chapter 5. Upgrading Server to Newer Version

General Notes on Upgrade

• Upgrade of CloverETL Server requires down time; plan a maintenance window

• Successful upgrade requires about 30 minutes; rollback requires 30 minutes

• Performing the below steps in development/testing environment first before moving onto production one

Upgrade Prerequisites

• Having new CloverETL Server web application archive ( clover.war

appropriate for the application server used) & license files available

• Having release notes for particular CloverETL version available (and all version between current and intended version to be upgraded to)

• Having the graphs and jobs updated and tested with regards to Known Issues & Compatibility for particular

CloverETL version.

• Having the CloverETL Server configuration properties file externalized from default location, see Chapter 6,

Configuration Sources and Their Priorities (p. 43)

• Standalone database schema where CloverETL Server stores configuration, see Chapter 8, Examples of DB

Connection Configuration (p. 53)

• Having a separate sandbox with test graph that can be run anytime to verify that CloverETL Server runs correctly and allows for running jobs

Upgrade Instructions

1. Suspend all sandboxes, wait for running graphs to finish processing

2. Shutdown CloverETL Server application (or all servers, if they run in cluster mode)

3. Backup existing CloverETL database schema (if any changes to the database schema are necessary, the new server will automatically make them when you start it for the first time)

4. Backup existing CloverETL web application archive ( clover.war

) & license files (on all nodes)

5. Backup existing CloverETL sandboxes (on all nodes)

6. Re-deploy the CloverETL Server web application. Instructions how to do that are application server dependent

- see Enterprise Server (p. 11) for installation details on all supported application servers. After you re-deploy,

your new server will be configured based on the previous version's configuration.

7. Replace old license files by the valid one (or you can later use web GUI form to upload new license). The license file is shipped as a text containing a unique set of characters. If you:

• received the new license as a file (

*.dat

), then simply use it as new license file.

• have been sent the licence text e.g inside an e-mail, then copy the license contents (i.e. all text between

Company

and

END LICENSE

) into a new file called clover-license.dat

. Next, overwrite the old license file with the new one or upload it in the web GUI.

See Installation of CloverETL Server License (p. 28) for details on license installation.

8. Start CloverETL Server application (on all nodes)

Chapter 5. Upgrading

Server to Newer Version

9. Review that contents of all tabs in CloverETL Server Console, especially scheduling and event listeners looks

OK.

10.Update graphs to be compatible with the particular version of CloverETL Server (this should be prepared and tested in advance)

11.Resume the test sandbox and run a test graph to verify functionality

12.Resume all sandboxes

Rollback Instructions

1. Shutdown CloverETL Server application

2. Restore CloverETL Server web application ( clover.war

) & license files (on all nodes)

3. Restore CloverETL Server database schema

4. Restore CloverETL sandboxes (on all nodes)

5. Start CloverETL Server application (on all nodes)

6. Resume the test sandbox and run a test graph to verify functionality

7. Resume all sandboxes

Important

Evaluation Version - a mere upgrade of your license is not sufficient. When moving from evaluation to enterprise server, you should not use the default configuration and database. Instead, take some time to configure Clover Server so that it best fits your production environment.

Part III. Configuration

We recommend the default installation (without any configuration) only for evaluation purposes. For production use, we recommend configuring a dedicated database and properly configuring the SMTP server for sending notifications.

Chapter 6. Configuration Sources and Their

Priorities

Configuration Sources

There are several sources of configuration properties. If a property isn't set, application default is used.

Configuration properties can be encrypted (see details in Chapter 10, Secure Configuration Properties (p. 76)).

Warning: Do not combine sources specified below. Configuration becomes confusing and maintenance will be much more difficult.

Environment Variables

Environment variables are variables configured by means of your operating system. E.g.

$PATH

is an environment variable.

Set environment variable with prefix clover.

, i.e. ( clover.config.file

)

Some operating systems may not use dot character, so also underlines (

) may be used instead of dots (

). So the clover_config_file

works as well.

System Properties

System properties are configured by means of JVM, e.g. with

-D

argument (

-Dclover.config.file

Set system property with clover.

prefix, i.e. ( clover.config.file

)

Also underlines (

) may be used instead of dots (

) so the clover_config_file

works as well.

Properties File on Default Location

Source is a common properties file (text file with key-value pairs):

[property-key]=[property-value]

By default, CloverETL tries to find the config file

[workingDir]/cloverServer.properties

Properties File on Specified Location

A file has the same file structure as in case above, but its location is specified with a clover_config_file or clover.config.file

environment variable or system property.

This is the recommended way of configuration if context parameters cannot be set in application server.

Modification of Context Parameters in web.xml

Unzip clover.war

and modify the

WEB-INF/web.xml

file. Add the following piece of code into the file:

<context-param>

<param-name>[property-name]</param-name>

<param-value>[property-value]</param-value>

</context-param>

Chapter 6. Configuration

Sources and Their Priorities

This way isn't recommended, but it may be useful when none of approaches above ways is possible.

Context Parameters (Available on Apache Tomcat)

Some application servers allow you to set context parameters without modification of a WAR file.

This way of configuration is possible, but is it not recommended, as Apache Tomcat may ignore some of context parameters in some environments. Use of a properties file is almost as convenient and much more reliable way.

Example for Apache Tomcat

On Tomcat, it is possible to specify context parameters in a context configuration file.

[tomcat_home]/conf/

Catalina/localhost/clover.xml

which is created automatically just after deployment of a CloverETL

Server web application.

You can specify a property with adding this element:

Priorities of Configuration Sources

Configuration sources have these priorities:

1. context parameters (specified in an application server or directly in a web.xml

file)

2. external config file; CS tries to find it in this order. Only one of them is loaded.

• path specified with a config.file

context parameter

• path specified with a clover_config_file or clover.config.file

system property.

• path specified with a clover_config_file or clover.config.file

environment variable.

• default location ( [workingDir]/cloverServer.properties

)

3. system properties

4. environment variables

5. default values

Chapter 7. Setup

CloverETL Server Setup helps you with configuration of CloverETL server. Instead of typing the whole configuration file in a text editor, the Setup generates content of the configuration file according to your instructions. It let you set up License and configure Database Connection, LDAP Connection, SMTP Server

Connection, Sandbox Paths, Encryption and Cluster Configuration.

The Setup is accessible from Server Console under Configuration

→

Setup.

Using Setup

If you start an unconfigured server, you can see decorators pointing to the Setup. The decorators mark problems to be solved. The displayed number corresponds to the number of items.

•

The following states mean error as mentioned in the text above:

error

warning

• restart required

Setup will help you with solving the problems.

Path to the Configuration File

Firstly, you have to configure a path to a configuration file. Without this, the Setup does not know, to which file the configuration should be saved. Each application server has different way to configure it.

Apache Tomcat

Edit bin/setenv.sh

(or bin/setenv.bat

) and add

-Dclover.config.file=/absolute/path/ to/cloverServer.properties

CATALINA_OPTS

Adding Libraries to Classpath

Secondly, you should configure connection to a database.

Place necessary libraries to a suitable directory. You usually need a jdbc driver for connection to a database or a .jar file with an encryption provider.

Having added the libraries restart the application server and configure Clover Server using Setup.

Chapter 7. Setup

Configuring Particular Items

Use Setup. Items configured in Setup are saved into a file defined with clover.config.file

If you need encryption, configure the Encryption first.

Configure connection to database and then update license. Later, you can configure other setup items.

Some Setup items (Database and Cluster) require restart of an application server. The latter items (License,

Sandboxes, E-mail, LDAP) do not need restarting; the changes to them are applied immediately. If you change something in a Configuration file tab, restart it or not depending on the updated part of the file.

As the last step, restart the server to use the new configuration.

Setup Tabs

Each setup page consists of a menu with setup tabs on the left, a main configuration part in the middle and a configuration status and a text on the right side.

Menu tabs have icons surrounding the text: a tick marks a configured tab, a wheel marks an inactive tab, a floppy disk marks a configuration that needs saving. Arrows signalize request on restart.

The main configuration part contains several buttons:

Discard Changes discards unsaved changes and returns to currently used values.

Save checks the configuration. If the configuration is valid, it is saved to the configuration file. If the configuration is not valid, Save Anyway button appears. The Save Anyway button allows you to save the configuration considered as invalid. E.g. Database connection is considered invalid if there is a required library missing. If you see Save disabled, use Validate to validate the configuration first.

Validate validates the configuration on a current tab.

Configuration File

Configuration tab displays content of a configuration file. You do not have to edit the content of the file manually, use the particular tab to configure the corresponding subsystem.

License

License tab let you specify license. The license is stored in database.

You should configure database before specifying license. Otherwise, you will have to specify the license twice.

Chapter 7. Setup

Database

Database tab let you configure connection to database. You can connect via JDBC.

Or you can use JNDI to access the datasource on an application server level. Choose a suitable item of a JNDI tree.

Sandboxes

Sandboxes let you configure path to sandboxes: shared, local, partitioned.

Chapter 7. Setup

Encryption

Encryption tab let you enable encryption of sensitive items of the configuration file. You can choose an encryption provider and an encryption algorithm. An alternative encryption provider can be used; the libs have to be added to classpath . (In the same way as database libraries.)

Save & Encrypt button saves the configuration and encrypts the passwords.

E-Mail

E-mail tab let you configure connection to SMTP server. The connection is necessary for reporting event on server via e-mails.

E-mail configuration can be tested with sending an e-mail from the dialog.

Chapter 7. Setup

LDAP

LDAP tab let you use an existing LDAP database for user authentication.

Chapter 7. Setup

Firstly, you should specify connection to the LDAP server. Secondly, define pattern for user DN. The login can be validated using any user matching the pattern.

Cluster

Cluster tab let you configure clustering features.

Chapter 7. Setup

Note

You can use the setup in a fresh installation of CloverETL Server, even if it had not been activated yet: log in into Server Console and use Close button to access the menu.

To access Setup section, you need a Server Setup permission. See Server Setup permission (p. 110).

Chapter 8. Examples of DB Connection

Configuration

In standalone deployment (non-clustered), configuration of DB connection is optional, since embedded Apache

Derby DB is used by default and it is sufficient for evaluation. However, configuration of external DB connection is strongly recommended for production deployment. It is possible to specify common JDBC DB connection attributes (URL, username, password) or JNDI location of DB DataSource.

In clustered deployment, at least one node in cluster must have DB connection configured. Other nodes may have their own direct connection (to the same DB) or may use another node as proxy for persistent operations, however

scheduler is active only on nodes with direct connection. See Part VI, “ Cluster ” (p. 228) for details about this

feature, this section describes only direct DB connection configuration.

DB Configurations and their changes may be as follows:

•

Embedded Apache Derby (p. 54)

•

MySQL (p. 55)

•

DB2 (p. 56)

•

Oracle (p. 59)

•

Microsoft SQL Server (p. 60)

•

Postgre SQL (p. 62)

•

JNDI DB DataSource (p. 63)

See Configuration Repository (p. 6) for officially supported versions of particular databases.

Chapter 8. Examples of DB

Connection Configuration

Embedded Apache Derby

Apache Derby embedded DB is used with default CloverETL Server installation. It uses working directory as a storage directory for data persistence by default. This may be a problem on some systems. In case of any problems with connecting to Derby DB, we recommend you configure connection to external DB or at least specify Derby home directory:

Set derby.system.home

system property to set path which is accessible for application server. You can specify this system property with this JVM execution parameter:

-Dderby.system.home=[derby_DB_files_root]

If you use a properties file for configuration, specify these parameters: jdbc.driverClassName, jdbc.url, jdbc.username, jdbc.password, jdbc.dialect.

For example: jdbc.driverClassName=org.apache.derby.jdbc.EmbeddedDriver

jdbc.url=jdbc:derby:databases/cloverDb;create=true jdbc.username= jdbc.password= jdbc.dialect=com.cloveretl.server.dbschema.DerbyDialect

Take a closer look at the jdbc.url

parameter. Part databases/cloverDb means a subdirectory for DB data. This subdirectory will be created in the directory which is set as derby.system.home

(or in the working directory if derby.system.home

is not set). Value databases/cloverDb is a default value, you may change it.

Derby JDBC 4 compliant driver is bundled with CloverETL Server, thus there is no need to add it on the classpath.

Chapter 8. Examples of DB

Connection Configuration

MySQL

CloverETL Server supports MySQL 5, up to version 5.5 included.

Creating database

The following steps will create database clover_db

and user clover

with password clover

1. Create database clover_db

, set charset and collate.

CREATE SCHEMA clover_db CHARACTER SET utf8 COLLATE utf8_unicode_ci;

2. Use clover_db as the current database.

USE clover_db;

3. Create a new user with password and host.

CREATE USER 'clover'@'%' IDENTIFIED BY 'clover';

4. Add all privileges to user 'clover' in DB clover_db.

GRANT ALL ON clover_db.* TO 'clover'@'%';

5. Reload privileges.

FLUSH privileges;

CloverETL setup

If you use a properties file for configuration, specify these parameters: jdbc.driverClassName, jdbc.url, jdbc.username, jdbc.password, jdbc.dialect.

For example: jdbc.driverClassName=com.mysql.jdbc.Driver

jdbc.url=jdbc:mysql://127.0.0.1:3306/clover?useUnicode=true&characterEncoding=utf8 jdbc.username=root jdbc.password= jdbc.dialect=org.hibernate.dialect.MySQLDialect

Please don't forget to add a JDBC 4 compliant driver on the classpath. A JDBC Driver which doesn't meet JDBC

4 won't work properly.

Create DB with a proper charset, like this:

CREATE DATABASE IF NOT EXISTS clover DEFAULT CHARACTER SET 'utf8';

Chapter 8. Examples of DB

Connection Configuration

DB2

Creating database

1. Create a dedicated user for the Clover database and set a password (UNIX/Linux).

useradd clover passwd clover

2. Create a new database.

db2 "CREATE DATABASE cloverdb PAGESIZE 32768 RESTRICTIVE"

3. Activate the database.

db2 activate db cloverdb

4. Connect to the database.

db2 connect to cloverdb

5. Grant the user DBADM authority (DBADM authority is an administrative authority for a specific database. The database administrator possesses the privileges that are required to create objects and issue database commands.

By default, DATAACCESS and ACCESSCTRL authority are also granted).

db2 "GRANT DBADM ON DATABASE TO USER clover"

6. Disconnect from database db2 connect reset

DB2 on Linux/Windows

If you use a properties file for configuration, specify these parameters: jdbc.driverClassName, jdbc.url, jdbc.username, jdbc.password, jdbc.dialect.

For example: jdbc.driverClassName=com.ibm.db2.jcc.DB2Driver

jdbc.url= jdbc:db2://localhost:50000/clover jdbc.username=usr jdbc.password=pwd jdbc.dialect=org.hibernate.dialect.DB2Dialect

Please don't forget to add a JDBC 4 compliant driver on the classpath. A JDBC Driver which doesn't meet JDBC

4 won't work properly.

Chapter 8. Examples of DB

Connection Configuration

Possible problems

Wrong pagesize

Database clover has to be created with suitable

PAGESIZE

. DB2 has several possible values for this property:

4096, 8192, 16384 or 32768.

CloverETL Server should work on DB with

PAGESIZE

set to 16384 or 32768. If

PAGESIZE

value is not set properly, there should be error message in the log file after failed CloverETL Server startup:

ERROR:

DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=16384;

ROOT, DRIVER=3.50.152

SQLERRMC contains suitable value for

PAGESIZE

You can create database with proper

PAGESIZE

like this:

CREATE DB clover PAGESIZE 32768;

The table is in the reorg pending state

After some ALTER TABLE commands, some tables may be in "reorg pending state". This behaviour is specific for

DB2. ALTER TABLE DDL commands are executed only during the first start of new CloverETL Server version.

Error message for this issue may look like this:

Operation not allowed for reason code "7" on table "DB2INST2.RUN_RECORD"..

SQLCODE=-668, SQLSTATE=57016 or like this

DB2 SQL Error: SQLCODE=-668, SQLSTATE=57016, SQLERRMC=7;DB2INST2.RUN_RECORD, DRIVER=3.50.152

In this case "RUN_RECORD" is table name which is in "reorg pending state" and "DB2INST2" is DB instance name.

To solve this, go to DB2 console and execute command (for table run_record): reorg table run_record

DB2 console output should look like this: db2 => connect to clover1

Database Connection Information

Database server = DB2/LINUX 9.7.0

SQL authorization ID = DB2INST2

Local database alias = CLOVER1 db2 => reorg table run_record

DB20000I The REORG command completed successfully.

db2 => disconnect clover1

DB20000I The SQL DISCONNECT command completed successfully.

"clover1" is DB name

Chapter 8. Examples of DB

Connection Configuration

DB2 does not allow ALTER TABLE which trims DB column length.

This problem depends on DB2 configuration and we've experienced this only on some AS400s so far. CloverETL

Server applies set of DP patches during the first installation after application upgrade. Some of these patches may apply column modifications which trims length of the text columns. These changes never truncate any data, however DB2 does not allow this since it "may" truncate some data. DB2 refuses these changes even in DB table which is empty. Solution is, to disable the DB2 warning for data truncation, restart CloverETL Server which applies patches, then enable DB2 warning again.

DB2 on AS/400

The connection on AS/400 might be slightly different.

If you use a properties file for configuration, specify these parameters: jdbc.driverClassName, jdbc.url, jdbc.username, jdbc.password, jdbc.dialect. For example: jdbc.driverClassName=com.ibm.as400.access.AS400JDBCDriver

jdbc.username=javlin jdbc.password=clover jdbc.url=jdbc:as400://host/cloversrv;libraries=cloversrv;date format=iso jdbc.dialect=org.hibernate.dialect.DB2400Dialect

Use credentials of your OS user for jdbc.username

and jdbc.password

cloversrv

in jdbc.url

above is the name of the DB schema.

You can create schema in AS/400 console:

• execute command

STRSQL

(SQL console)

• execute

CREATE COLLECTION cloversrv IN ASP 1

• cloversrv

is the name of the DB schema and it may be at most 10 characters long

Proper JDBC driver must be in the application server classpath.

I use JDBC driver jt400ntv.jar

, which I've found in

/QIBM/ProdData/Java400

on the server.

Use jt400ntv.jar

JDBC driver.

Please don't forget to add a JDBC 4 compliant driver on the classpath. A JDBC Driver which doesn't meet JDBC

4 won't work properly.

Chapter 8. Examples of DB

Connection Configuration

Oracle

Creating database

Run the following script to create a role, a user and a tablespace for CloverETL server.

Role: cloverRole

User: cloverUser

Password: cloverPassword

-- Create a new role and grant it privileges

CREATE ROLE cloverRole NOT IDENTIFIED;

GRANT CREATE SESSION TO cloverRole;

GRANT ALTER SESSION TO cloverRole;

GRANT CREATE TABLE TO cloverRole;

GRANT CREATE SEQUENCE TO cloverRole;

GRANT CREATE TRIGGER TO cloverRole;

-- Create a new database user with password

CREATE USER cloverUser IDENTIFIED BY cloverPassword;

-- Set quota on tablespace

GRANT UNLIMITED TABLESPACE TO cloverUser;

-- Connect a new role to a new user

GRANT cloverRole TO cloverUser;

CloverETL setup

If you use a properties file for configuration, specify these parameters: jdbc.driverClassName, jdbc.url, jdbc.username, jdbc.password, jdbc.dialect.

For example: jdbc.driverClassName=oracle.jdbc.OracleDriver

Creating database

Advanced users can create their own table space

We are going to create a database for clover to use a 'user group' role which will own the database and a user role which we will add to the user group. This user role will be then used by the server to access the database.

Database name: clover_db

UserGroup: cloveretl

User: clover

Password: clover

1. Optionally, you can create new tablespace

2. Connect as postgres (default admin) to the default db postgres and execute the following commands:

CREATE ROLE cloveretl NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOLOGIN;

CREATE ROLE clover NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN ENCRYPTED PASSWORD 'clover';

GRANT cloveretl TO clover;

CREATE DATABASE clover_db;

GRANT ALL ON DATABASE clover_db TO cloveretl;

REVOKE ALL ON DATABASE clover_db FROM public;

To separate database into its own tablespace create a tablespace before creating the database: and use the following command to create the db:

CREATE DATABASE clover_db WITH OWNER cloveretl TABLESPACE tablespace_name; https://www.postgresql.org/docs/9.2/static/sql-createtablespace.html

CloverETL setup

If you use a properties file for configuration, specify these parameters: jdbc.driverClassName, jdbc.url, jdbc.username, jdbc.password, jdbc.dialect.

For example: jdbc.driverClassName=org.postgresql.Driver

jdbc.url=jdbc:postgresql://localhost/clover?charSet=UTF-8 jdbc.username=postgres jdbc.password= jdbc.dialect=org.hibernate.dialect.PostgreSQLDialect

Please don't forget to a add JDBC 4 compliant driver on the classpath. A JDBC Driver which doesn't meet the

JDBC 4 specification won't work properly.

The JDBC driver for PostgreSQL can be downloaded from: https://jdbc.postgresql.org/download.html. In Apache

Tomcat you would place libs into

$CATALINA_HOME/libs

directory.

Chapter 8. Examples of DB

Connection Configuration

JNDI DB DataSource

CloverETL Server can connect to database using JNDI DataSource, which is configured in application server or container. However there are some CloverETL parameters which must be set, otherwise the behaviour may be unpredictable: datasource.type=JNDI # type of datasource; must be set, because default value is JDBC datasource.jndiName=# JNDI location of DB DataSource; default value is java:comp/env/jdbc/clover_server # jdbc.dialect=# Set dialect according to DB which DataSource is connected to.

The same dialect as in sections above. #

The parameters above may be set in the same ways as other params (in properties file or Tomcat context file)

Example of DataSource configuration in Apache Tomcat. Add following context resource configuration may be added to the

[tomcat_home]/conf/server.xml

file to the

<Host>

element.

<Resource name="jdbc/clover_server" auth="Container"

type="javax.sql.DataSource" driverClassName="com.mysql.jdbc.Driver"

url="jdbc:mysql://127.0.0.1:3306/clover?useUnicode=true&characterEncoding=utf8"

username="root" password=""

maxActive="20" maxIdle="10" maxWait="-1"/>

</Context>

Do no put the code above into

element, since the resource would not be visible by the CloverETL webapp.

Note

The resource configuration may be also added to context file

[tomcat_home]/conf/

Catalina/localhost/clover.xml

Note

Special characters you type in the context file have to be specified as XML entities. E.g. ampersand

"&" as "&" etc.

Important

The default jndi pool DBCP in Apache Tomcat does not handle connections in efficient way. With

DBCP jndi pool, low performance can be seen if DBOutputTable with returning statement is used.

Use tomcat-jdbc-pool instead. Just add factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"

to the definition of the jndi resource. See https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html

See Chapter 9, List of Properties (p. 70) for list of properties.

Chapter 8. Examples of DB

Connection Configuration

Encrypted JNDI

You can store password for database connection encrypted. The configuration differs between particular application servers.

Encrypted JNDI on Tomcat (p. 64)

Encrypted JNDI on Jetty 9 (9.2.6) (p. 65)

Encrypted JNDI on JBoss 6.0.0

(p. 65)

JBoss 7 (p. 67)

Encrypted JNDI on Glassfish 3 (3.1.2.2) (p. 68)

Encrypted JNDI on WebSphere 8.5.5.0

(p. 68)

Encrypted JNDI on WebLogic (p. 69)

Encrypted JNDI on Tomcat

You need secure-cfg-tool to encrypt the passwords. Use the version of secure-cfg-tool

corresponding to the version of CloverETL Server. Usage of the tool is described in Chapter 10, Secure

Configuration Properties (p. 76).

Use encrypt.sh

or encrypt.bat

for encryption of a password. Place the encrypted password into a configuration file, and put cloveretl-secure-jndi-resource-{version}.jar

and jasypt-1.9.0.jar

files on classpath of the application server. The

.jar

files can be found in tomcatsecure-jndi-resource

directory packed in secure-cfg-tool.

The tomcat-secure-jndi-resource

directory contains useful file

README

with further details on encrypted JNDI.

Example of encrypted JNDI connection for Postgresql

Encrypt the password:

./encrypt.sh -a PBEWithSHA1AndDESede

The configuration is placed in

${CATALINA_HOME}/webapps/clover/META-INF/cotext.xml

. Note that the encryption algorithm PBEWithSHA1AndDESede is not default.

<Resource name="jdbc/clover_server"

auth="Container"

factory="com.cloveretl.secure.tomcatresource.SecureDataSourceFactory"

secureAlgorithm="PBEWithSHA1AndDESede"

type="javax.sql.DataSource"

driverClassName="org.postgresql.Driver"

url="jdbc:postgresql://127.0.0.1:5432/clover410m1?charSet=UTF-8"

username="conf#rPz5Foo7HPn4dFTRV5Ourg=="

password="conf#4KlNp8/FVDR+rTWX0dEqWA=="

maxActive="20"

maxIdle="10"

maxWait="-1"/>

If you use other JCE (e.g. Bouncy Castle), it has to be added to the classpath of an application server

(

${CATALINA_HOME}/lib

). The encrypt command requires the path to directory with JCE too.

./encrypt.sh -l ~/lib/ -c org.bouncycastle.jce.provider.BouncyCastleProvider

-a PBEWITHSHA256AND256BITAES-CBC-BC

<Resource name="jdbc/clover_server"

auth="Container"

factory="com.cloveretl.secure.tomcatresource.SecureDataSourceFactory"

secureProvider="org.bouncycastle.jce.provider.BouncyCastleProvider"

secureAlgorithm="PBEWITHSHA256AND256BITAES-CBC-BC"

Chapter 8. Examples of DB

Connection Configuration

type="javax.sql.DataSource"

driverClassName="org.postgresql.Driver"

url="jdbc:postgresql://127.0.0.1:5432/clover410m1?charSet=UTF-8"

username="conf#Ws9IuHKo9h7hMjPllr31VxdI1A9LKIaYfGEUmLet9rA="

password="conf#Cj1v59Z5nCBHaktn6Ubgst4Iz69JLQ/q6/32Xwr/IEE="

maxActive="20" maxIdle="10"

maxWait="-1"/>

Encrypted JNDI on Jetty 9 (9.2.6)

http://eclipse.org/jetty/documentation/current/configuring-security-secure-passwords.html

Configuration of a JNDI jdbc connection pool is stored in the plain text file,

$JETTY_HOME/etc/jetty.xml

<Arg>jdbc/MysqlDS</Arg>

<Arg>

<Set name="URL">jdbc:mysql://localhost:3306/clover_empty</Set>

<Set name="Password">password</Set>

</New>

</Arg>

</New>

Obfuscating the Password

Password can be obfuscated using org.eclipse.jetty.util.security.Password

class within lib/ jetty-util-{VERSION}.jar

: java -cp lib/jetty-util-9.2.6.v20141205.jar org.eclipse.jetty.util.security.Password password

Command returns obfuscated and hashed password. The obfuscated one will be used to replace the plain password value.

Replacing the Password

Replace the plain text password with the Call element. Its only argument is a string starting with the OBF: prefix returned by the command mentioned in the previous section.

<Arg>jdbc/MysqlDS</Arg>

<Arg>

<Set name="URL">jdbc:mysql://localhost:3306/clover_empty</Set>

</Call>

</Set>

</New>

</Arg>

</New>

Password in the JMS connection can be also obfuscated.

Encrypted JNDI on JBoss 6.0.0

Original datasource with an unencrypted password:

Chapter 8. Examples of DB

Connection Configuration

<local-tx-datasource>

<jndi-name>MysqlDS</jndi-name>

<connection-url>jdbc:mysql://127.0.0.1:3306/clover</connection-url>

<driver-class>com.mysql.jdbc.Driver</driver-class>

<user-name>user</user-name>

<password>password</password>

</local-tx-datasource>

</datasources>

Encrypt the data source password

Linux java -cp client/jboss-logging.jar:lib/jbosssx.jar org.jboss.resource.security.SecureIdentityLoginModule password

Windows java -cp client\jboss-logging.jar;lib\jbosssx.jar org.jboss.resource.security.SecureIdentityLoginModule password

NOTE: in the JBoss documentation client/jboss-logging-spi.jar

is used, but there is no such a file in my JBossAS [6.0.0.Final "Neo"], but client/jboss-logging.jar

can be used instead.

The command will return an encrypted password, e.g. 5dfc52b51bd35553df8592078de921bc.

Create a new application authentication policy in conf/login-config.xml

within currently used server's profile directory (e.g. server/default/conf/login-config.xml

<application-policy name="EncryptDBPassword">

<login-module code="org.jboss.resource.security.SecureIdentityLoginModule" flag="required">

<module-option name="username">user</module-option>

<module-option name="password">5dfc52b51bd35553df8592078de921bc</module-option>

<module-option name="managedConnectionFactoryName">jboss.jca:name=MysqlDS,service=LocalTxCM</module-option>

</login-module>

</authentication>

</application-policy>

Replace authentication entries with a reference to the application authentication policy

<security-domain>EncryptDBPassword</security-domain>

The final datasource looks like this:

<local-tx-datasource>

<jndi-name>MysqlDS</jndi-name>

<connection-url>jdbc:mysql://127.0.0.1:3306/clover</connection-url>

<driver-class>com.mysql.jdbc.Driver</driver-class>

<security-domain>EncryptDBPassword</security-domain>

</local-tx-datasource>

</datasources>

The same mechanism can be probably used also for JMS.

<tx-connection-factory>

...

<security-domain-and-application>RealmWithEncryptedPassword</security-domain-and-application>

Chapter 8. Examples of DB

Connection Configuration

...

</tx-connection-factory>

See http://docs.jboss.org/jbosssecurity/docs/6.0/security_guide/html/Encrypting_Data_Source_Passwords.html

JBoss 7

JBoss 7 - JBoss EAP 6.2.0.GA - AS 7.3.0.Final-redhat-14

Configuration steps are similar to configuring of JBoss 6.

All configuration takes place in the single configuration file, e.g. for standalone profile

JBOSS_HOME/ standalone/configuration/standalone.xml

Original datasource:

<connection-url>jdbc:mysql://localhost:3306/clover</connection-url>

<driver>mysql</driver>

<pool>

<max-pool-size>30</max-pool-size>

</pool>

<user-name>user</user-name>

<password>password</password>

</security>

</datasource>

<driver-class>com.mysql.jdbc.Driver</driver-class>

</driver>

</drivers>

In JBOSS_HOME directory run cli command: java -cp modules/system/layers/base/org/picketbox/main/picketbox-4.0.19.SP2-redhat-1.jar:client/jboss-logging.jar org.picketbox.datasource.security.SecureIdentityLoginModule password

The command will return an encrypted password, e.g. 5dfc52b51bd35553df8592078de921bc.

Add a new security-domain to security-domains, the password value is a result of the command from the previous step.

<security-domain name="EncryptDBPassword" cache-type="default">

<login-module code="org.picketbox.datasource.security.SecureIdentityLoginModule" flag="required">

<module-option name="username" value="user"/>

<module-option name="password" value="5dfc52b51bd35553df8592078de921bc"/>

<module-option name="managedConnectionFactoryName" value="jboss.jca:service=LocalTxCM,name=MysqlPool"/>

</login-module>

</authentication>

</security-domain>

Replace user and password with a reference to the security domain.

<connection-url>jdbc:mysql://localhost:3306/clover</connection-url>

<driver>mysql</driver>

<pool>

<max-pool-size>30</max-pool-size>

Chapter 8. Examples of DB

Connection Configuration

</pool>

<security-domain>EncryptDBPassword</security-domain>

</security>

</datasource>

<driver-class>com.mysql.jdbc.Driver</driver-class>

</driver>

</drivers>

</datasources>

The same mechanism can be probably used also for JMS.

http://middlewaremagic.com/jboss/?p=1026

Encrypted JNDI on Glassfish 3 (3.1.2.2)

Configuration of jdbc connection pool is stored in the plain text file, $DOMAIN/config/domain.xml.

<jdbc-connection-pool driver-classname="com.mysql.jdbc.Driver" datasource-classname="" res-type="java.sql.Driver" description="" name="jdbc/MysqlDS">

</jdbc-connection-pool>

Password is unencrypted, but can be replaced with so called password alias:

A password alias stores a password in encrypted form in the domain keystore, providing a clear-text alias name to use instead of the password. In password files and the domain configuration file, use the form ${ALIAS=aliasname} to refer to the encrypted password.

Creating a Password Alias

There are two ways to create a password alias: using create-password-alias command in a command-line adminconsole utility, or in the web Server Administration Console in the Password Aliases section (Domain->Password

Aliases).

Replacing the Password with the Password Alias

Replace password (the attribute value) with a ${ALIAS=password_alias_name} string, where password_alias_name is the name of the alias.

<jdbc-connection-pool driver-classname="com.mysql.jdbc.Driver" datasource-classname="" res-type="java.sql.Driver" description="" name="jdbc/MysqlDS">

</jdbc-connection-pool>

NOTE: Glassfish's Administration Server Console mentions a lower case keyword alias, but it doesn't work for me. Changing to upper case ALIAS makes the connection pool work.

Password for a JMS connection can be replaced with an alias as well.

Encrypted JNDI on WebSphere 8.5.5.0

In WebSphere, user credentials aren't saved in plain text, but as J2C authentication data.

http://www-01.ibm.com/support/docview.wss?uid=nas8N1011315

Chapter 8. Examples of DB

Connection Configuration

The same mechanism can be used also for JMS connection.

(Configuring an external JMS provider: https://www.ibm.com/developerworks/community/blogs/timdp/entry/ using_activemq_as_a_jms_provider_in_websphere_application_server_7149?lang=en )

Encrypted JNDI on WebLogic

Password in a JNDI datasource file is encrypted by default when created by admin's web console (Service/

Datasource).

Example of datasource file (located in DOMAIN/config/jdbc/ directory):

<?xml version='1.0' encoding='UTF-8'?>

<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source" xmlns:sec="http://xmlns.oracle.com/weblogic/security" xmlns:wls="http://xmlns.oracle.com/weblogic/security/wls" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/jdbc-data-source http://xmlns.oracle.com/weblogic/jdbc-data-source/1.0/jdbc-data-source.xsd">

<name>MysqlDS</name>

<jdbc-driver-params>

<url>jdbc:mysql://127.0.0.1:3306/clover</url>

<driver-name>com.mysql.jdbc.Driver</driver-name>

</property>

</properties>

<password-encrypted>{AES}zIiq6/JutK/wD4CcRPX1pOueIlKqc6uRVxAnZZcC3pI=</password-encrypted>

</jdbc-driver-params>

<jdbc-connection-pool-params>

<test-table-name>SQL SELECT 1</test-table-name>

</jdbc-connection-pool-params>

<jdbc-data-source-params>

<jndi-name>jdbc/MysqlDS</jndi-name>

<global-transactions-protocol>OnePhaseCommit</global-transactions-protocol>

</jdbc-data-source-params>

</jdbc-data-source>

The same mechanism is also used for encrypting password in the JMS connection.

(Configuring an external JMS provider: http://docs.oracle.com/cd/E12840_01/wls/docs103/ConsoleHelp/ taskhelp/jms_modules/foreign_servers/ConfigureForeignServers.html

Chapter 9. List of Properties

Table 9.1. General configuration

config.file

license.file

key

engine.config.file

sandboxes.home

description

location of a CloverETL Server configuration file

default

[working_dir]/ cloverServer.properties

location of a CloverETL Server licence file (license.dat) location of a CloverETL engine configuration properties file properties file packed with CloverETL

This property is primarily intended to be used as a placeholder in the sandbox root path specification. So the sandbox path is specified with the placeholder and it's resolved to the real path just before it's used.

Sandbox path may still be specified as an absolute path, but placeholder has some significant advantages:

${user.data.home}/

CloverETL/ sandboxes

* sandbox definition may be exported/imported to another environment with a different directory structure

* user creating sandboxed doesn't have to care about physical location on the filesystem

* each node in cluster environment may have a different

"sandboxes.home" value, so the directory structure doesn't have to be identical private.properties

The default value uses content of a "user.data.home" configuration property, which points to the home directory of the user which runs the JVM process.

Directory depends on the OS. On unix-like systems it's typically /home/[username]

List of server properties which are used only by

CloverETL Server code. So these properties are not accessible outside of the ServerFacade. By default there are all properties which may contain password in the list. Basically it means, that their values are not visible for web GUI users. Values are replaced by single star

"*". Changes in this list may cause unexpected behavior of some server API.

jdbc.password, executor.password, security.ldap.password, clover.smtp.password

engine.plugins.additional.src

This property may contain an absolute path to some

"source" of additional CloverETL engine plugins.

These plugins are not a substitute for plugins packed in WAR. "Source" may be a directory or a zip file.

Both, a directory and a zip, must contain a subdirectory for each plugin. Changes in the directory or the ZIP file apply only when the server is restarted. For details

see Chapter 27, Extensibility - CloverETL Engine

Plugins (p. 226).

empty datasource.type

Set this explicitly to JNDI if you need CloverETL

Server to connect to a DB using JNDI datasource. In such case, "datasource.jndiName" and "jdbc.dialect" parameters must be set properly. Possible values: JNDI

| JDBC

JDBC

Chapter 9. List of Properties

key

datasource.jndiName

description default

JNDI location of a DB DataSource. It is applied only if

"datasource.type" is set to "JNDI".

java:comp/env/jdbc/ clover_server class name for jdbc driver name jdbc.driverClassName

jdbc.url

jdbc.username

jdbc.password

jdbc.dialect

jdbc url used by CloverETL Server to store data jdbc database user name jdbc database user name hibernate dialect to use in ORM quartz.driverDelegateClass

SQL dialect for quartz. Value is automatically derived from "jdbc.dialect" property value.

sandboxes.access.check.boundaries.enabled

true true | false If it is set to false, then path relative to sandbox root may point out of the sandbox. No file/ folder outside of the sandbox is accessible by the relative path otherwise.

security.session.validity

Session validity in milliseconds. When the request of logged-in user/client is detected, validity is automatically prolonged.

14400000 security.session.exchange.limit Interval for exchange of invalid tokens in milliseconds. 360000 security.default_domain

Domain which all new users are included in. Stored in user's record in the database. Shouldn't be changed unless the "clover" must be white-labelled.

clover security.basic_authentication.features_list

List of features which are accessible using HTTP and which should be protected by Basic HTTP

Authentication. The list has form of semi-colon separated items; Each feature is specified by its servlet path.

/request_processor;/ simpleHttpApi;/ launch;/launchIt;/ downloadStorage;/ downloadFile;/ uploadSandboxFile;/ downloadLog;/ webdav security.basic_authentication.realm

CloverETL Server

Realm string for HTTP Basic Authentication.

security.digest_authentication.features_list

List of features which are accessible using HTTP and which should be protected by HTTP Digest

Authentication. The list has form of semi-colon separated items. Each feature is specified by its servlet path.

Please keep in mind, that HTTP Digest Authentication is feature added to the version 3.1. If you upgraded your older CloverETL Server distribution, users created before the upgrade cannot use the HTTP Digest

Authentication until they reset their passwords. So when they reset their passwords (or the admin does it for them), they can use Digest Authentication as well as new users.

security.digest_authentication.storeA1.enabled

false

Chapter 9. List of Properties

key description

Switch whether the A1 Digest for HTTP Digest

Authentication should be generated and stored or not. Since there is no CloverETL Server API using the HTTP Digest Authentication by default, it's recommended to keep it disabled. This option is not automatically enabled when any feature is specified in the security.digest_authentication.features_list

property.

default

security.digest_authentication.realm

Realm string for HTTP Digest Authentication. If it is changed, all users have to reset their passwords, otherwise they won't be able to access to the server features protected by HTTP digest Authentication.

CloverETL Server security.digest_authentication.

nonce_validity clover.event.

fileCheckMinInterval

Interval of validity for HTTP Digest Authentication specified in seconds. When the interval passes, server requires new authentication from the client. Most of the

HTTP clients do it automatically.

300

Interval of the timer, running file event listener checks

(in milliseconds). See File Event Listeners (remote and local) (p. 194) for details.

1000 clover.smtp.transport.protocol

SMTP server protocol. Possible values are "smtp" or

"smtps".

smtp clover.smtp.host

SMTP server hostname or IP address clover.smtp.port

clover.smtp.authentication

clover.smtp.username

clover.smtp.password

clover.smtp.additional.*

SMTP server port true/false If it is false, username and password are ignored

SMTP server username

SMTP server password logging.project_name

Properties with a "clover.smtp.additional." prefix are automatically added (without the prefix) to the

Properties instance passed to the Mailer. May be useful for some protocol specific parameters. The prefix is removed.

used in log messages where it is necessary to name the product name

CloverETL logging.default_subdir

name of a default subdirectory for all server logs; it is relative to the path specified by system property

"java.io.tmpdir". Don't specify as an absolute path, use properties which are intended for absolute path.

cloverlogs logging.logger.server_audit.enabled

Enables logging of operations called on ServerFacade and JDBC proxy interfaces. The name of output file is

"server-audit.log". It is stored in the same directory as other CloverETL Server log files by default. The default logging level is DEBUG so it logs all operations which may process any change.

false launch.log.dir

Location, where server should store launch requests

logs. See Launch Services (p. 216) for details.

${java.io.tmpdir}/

[logging.

default_subdir]/ launch where

Chapter 9. List of Properties

key

graph.logs_path

description default

${java.io.tmpdir} is system property

Location, where server should store Graph run logs. See

Chapter 11, Logging (p. 80) for details.

${java.io.tmpdir}/

[logging.

default_subdir]/ graph where

${java.io.tmpdir} is system property

%d %-5p

%-3X{runId} [%t]

%m%n

UTF-8 logging.appender.jobs.encoding Encoding of the jobs' log files temp.default_subdir

graph.pass_event_params

_to_graph_in_old_style

Name of default subdirectory for server tmp files; it is relative to the path specified by system property

"java.io.tmpdir".

clovertmp

Since 3.0. It is switch for backwards compatibility of passing parameters to the graph executed by a graph event. In version prior to 3.0 all params has been passed to executed graph. Since 3.0 just specified parameters are passed. Please see

Start a Graph (p. 152) for

details.

false threadManager.pool.corePoolSize

Number of threads which are always active (running or idling). Related to a thread pool for processing server events.

threadManager.pool.queueCapacity

Max size of the queue (FIFO) which contains tasks waiting for an available thread. Related to a thread pool for processing server events. For queueCapacity=0, there are no waiting tasks, each task is immediately executed in an available thread or in a new thread.

threadManager.pool.maxPoolSize

8192

Max number of active threads. If no thread from a core pool is available, the pool creates new threads up to

"maxPoolSize" threads. If there are more concurrent tasks then maxPoolSize, thread manager refuses to execute it.

threadManager.pool.allowCoreThreadTimeOut

false

Switch for idling threads timeout. If true, the

"corePoolSize" is ignored so all idling threads may be time-outed threadManager.pool.keepAliveSeconds

20 task.archivator.batch_size

launch.http_header_prefix

timeout for idling threads in seconds

Max number of records deleted in one batch. It is used for deleting of archived run records.

Prefix of HTTP headers added by launch services to the

HTTP response.

X-cloveretl

Chapter 9. List of Properties

key

task.archivator.

archive_file_prefix license.context_names

description

Prefix of archive files created by the archivator.

default

cloverArchive_

A comma-separated list of web-app contexts which may contain license. Each of them has to start with a slash!

Works only on Apache Tomcat.

/clover-license,/ clover_license properties_resolver.resolve_server_props.server_props_list_additional

If already the property specified by to resolve the is property properties_resolver.resolve_server_props.server_props_list_default, don't add it to this property.

properties_resolver.resolve_server_props.server_props_list_default

A list of properties from a subset of properties, which values are resolved. The properties' values may use system properties or environment variables as placeholders. Values are resolved during the server startup. If the system property is changed later, the resolved CloverETL Server property value doesn't change. Users are discouraged from modification sandboxes.home, sandboxes.home.local, sandboxes.home.partitioned, cluster.jgroups.bind_address, cluster.jgroups.start_port, cluster.jgroups.external_address, cluster.jgroups.external_port, cluster.jgroups.tcpping.initial_hosts, of the property, unles it's necessary. Instead users may add more properties by modifying property: cluster.group.name, cluster.http.url

properties_resolver.resolve_server_props.server_props_list_additional

properties_resolver.placeholders.server_props_list_default

A list of properties from a subset of properties, that may be used as placeholders and shall be resolved if used in paths. The properties can be used if you define a path to the root of sandbox, or to locations of local or partitioned sandboxes, or path to a script, or path in archiver job. Users are strongly discouraged from modification of the property. The property name changed since CloverETL 4.2, however also the obsolete name is still accepted to maintain backwards compatibility.

sandboxes.home, sandboxes.home.local, sandboxes.home.partitioned, user.data.home

webDav.method.propfind.maxDepth

Maximum depth for webDAV method PROPFIND.

When the depth is not specified, the default is supposed to be infinite (according to the rfc2518), however it's necessary to set some limit, otherwise the webDav client might overload the server filesystem.

Also if the depth value specified by webDAV client in the request is higher than the pre-configured max depth, only the pre-configured maximum is used.

jvm.implementation.check.enabled

warnings when unsupported Java implementation is used.

true

Chapter 9. List of Properties

Table 9.2. Defaults for job execution configuration - see

Job Config Properties

(p. 119) for details

key

executor.tracking_interval

executor.log_level

description default

An interval in milliseconds for scanning of a current status of a running graph. The shorter interval, the bigger log file.

2000

Log level of graph runs. TRACE | DEBUG | INFO | WARN |

ERROR

INFO executor.max_job_tree_depth

Defines maximal depth of the job execution tree, e.g. for recursive job it defines the maximal level of recursion (counting from root job).

32 executor.max_running_concurrently Amount of graph instances which may exist (or run) concurrently.

0 means no limits

0 executor.max_graph_instance_age

Interval in milliseconds. Specifies how long graph instance can be idling before it is released from memory. 0 means no limits.

0 executor.classpath

executor.skip_check_config

executor.password

executor.verbose_mode

executor.use_jmx

executor.debug_mode

This property has been renamed since 2.8. Original name was executor.maxGraphInstanceAge

Classpath for transformation/processor classes used in the graph.

Directory [sandbox_root]/trans/ does not have to be listed here, since it is automatically added to graph run classpath.

Disables check of graph configuration. Increases performance of graph execution, however may be useful during graph development.

true

This property is deprecated. Password for decoding of encoded

DB connection passwords.

If true, more descriptive logs of graph runs are generated.

If true, graph executor registers jmx mBean of running graph.

true true

If true, edges with enabled debug store data into files in debug directory.

false

See Chapter 29, Clustering Features (p. 229) for more properties.

Chapter 10. Secure Configuration Properties

Some configuration properties can be confidential (e.g. password to database, mail client) and thus it's desirable to encrypt them. For this purpose there is a command-line utility secure-cfg-tool.jar.

Basic Utility Usage (p. 76)

Advanced Usage - Custom Settings (p. 77)

Basic Utility Usage

1. Get a utility archive file ( secure-cfg-tool.zip

) and unzip it.

The utility is available in the download section of your CloverETL account - at the same location as the download of CloverETL Server.

2. Execute the script given for your operating system, encrypt.bat

for MS Windows, encrypt.sh

for

Linux. You will be asked for inserting a value of a configuration property intended to be encrypted.

Example:

C:\secure-cfg-tool>encrypt.bat

**************************************************************

Secure config encryption (use --help or -h option to show help)

**************************************************************

****** Config settings ******

Provider: SunJCE

Algorithm: PBEWithMD5AndDES

*****************************

Enter text to encrypt: mypassword

Text to encrypt: "mypassword"

Encrypted text: conf#eCflGDlDtKSJjh9VyDlRh7IftAbI/vsH

C:\secure-cfg-tool>

If you want configure the way how are values encrypted, see Advanced Usage - Custom Settings (p. 77)

3. Encrypted string has conf#encrypted_property format. The encrypted string can be used as a value of a configuration property in the properties file, clover.xml

file or web.xml

file (see details about

configuration sources in Chapter 6, Configuration Sources and Their Priorities (p. 43)).

Example (snippet of a configuration property file): jdbc.driverClassName=com.mysql.jdbc.Driver

jdbc.url=jdbc:mysql://example.com:3306/clover?useUnicode=true&characterEncoding=utf8 jdbc.username=example jdbc.password=conf#eCflGDlDtKSJjh9VyDlRh7IftAbI/vsH jdbc.dialect=org.hibernate.dialect.MySQLDialect

Note

Alternatively, java -jar secure-cfg-tool.jar command can be used.

Chapter 10. Secure

Configuration Properties

Important

Values encrypted by a Secure parameter form (Chapter 13, Secure Parameters

be used as a value of a configuration property.

Advanced Usage - Custom Settings

The way how configuration values are encrypted described so far, uses default configuration settings (a default provider and algorithm). But if there is a need to change these default settings with the custom ones, the secure-

cfg-tool.jar utility offers a set of parameters to achieve that.

Table 10.1. Parameters

Parameter

--algorithm, -a

--file, -f

--help, -h

--providerclass, -c

--providerlocation, -l

--providers, -p

Description

algorithm to encrypt config file location show help

Example

--algorithm PBEWithMD5AndDES

-f C:\User\John\cloverServer.properties

--help custom provider class -c org.provider.ProviderClass

path to jar/folder containing custom provider class (it will be added to classpath)

--providerlocation C:\User\John\lib

\customprovider.jar, -l C:\User\John\lib\ print available security providers and their algorithms

--providers

Note

To demonstrate usage of an external provider the Bouncy Castle provider is used.

To find out a list of algorithms use -p or --providers

C:\secure-cfg-tool>encrypt.bat -p

If you want to find out a list of algorithms of an external provider, you must pass the provider's class name and path to jar file(s)

C:\secure-cfg-tool>encrypt.bat -p -c org.bouncycastle.jce.provider.BouncyCastleProvider -l C:\User\John\bcprov-jdk15on-152.jar

Result might look like this

***** List of available providers and their algorithms *****

Provider: SunJCE

Provider class: com.sun.crypto.provider.SunJCE

Algorithms:

PBEWithMD5AndDES

PBEWithSHA1AndDESede

PBEWithSHA1AndRC2_40

Provider: BC

Provider class: org.bouncycastle.jce.provider.BouncyCastleProvider

Algorithms:

PBEWITHMD2ANDDES

PBEWITHMD5AND128BITAES-CBC-OPENSSL

PBEWITHMD5AND192BITAES-CBC-OPENSSL

PBEWITHMD5AND256BITAES-CBC-OPENSSL

Chapter 10. Secure

Configuration Properties

Provider class is displayed on the row starting with Provider class, algorithms are strings with PBE prefix. Both can be used to configure encryption.

Configuring the Encryption Process

Algorithm and provider can be passed to the utility in two ways.

•

Using command line arguments

To change the algorithm use argument -a. Provider remains the default one (SunJCE in case of Oracle Java).

C:\secure-cfg-tool>encrypt.bat -a PBEWithMD5AndDES

Using of an external provider is a little more complex. Provider's class name must be specified (argument

--providerclass or -c) and jar(s) must be added to the classpath (argument --providerlocation, -l). Provider location must point to concrete jar file or directory containing jar(s) and can be used several times for several paths.

C:\secure-cfg-tool>encrypt.bat -a PBEWITHSHA256AND256BITAES-CBC-BC -c org.bouncycastle.jce.provider.BouncyCastleProvider -l C:\User\John\bcprov-jdk15on-152.jar

•

Using configuration file

A configuration file is a common properties file (text file with key-value pairs):

[property-key]=[property-value]

It might look like this (example comes from secure.config.example.properties

, distributed within secure-cfg-tool.zip

security.config_properties.encryptor.providerClassName=org.bouncycastle.jce.provider.BouncyCastleProvider

security.config_properties.encryptor.algorithm=PBEWITHSHA256AND256BITAES-CBC-BC

security.config_properties.encryptor.provider.location=C:\\User\\libs

To let utility know about the configuration file use -f argument

C:\secure-cfg-tool>encrypt.bat -f secure.config.example.properties

Note

More jar locations can be set in the security.config_properties.encryptor.providerLocation, locations are delimited by semicolon.

Configuring an application server

CloverETL Server application needs to know how the values have been encrypted, therefore the properties must

be passed to the server (see details in Part III, “ Configuration ” (p. 42)). For example:

Chapter 10. Secure

Configuration Properties

...

security.config_properties.encryptor.providerClassName=org.bouncycastle.jce.provider.BouncyCastleProvider

security.config_properties.encryptor.algorithm=PBEWITHSHA256AND256BITAES-CBC-BC

...

Important

If a third-party provider is used, its classes must be accessible for the application server. Property

security.config_properties.encryptor.providerLocation will be ignored.

Chapter 11. Logging

Main Logs (p. 80)

Another Useful Logging Settings (p. 80)

Access Log in Apache Tomcat (p. 81)

Graph Run Logs (p. 81)

Server Audit Logs (p. 81)

Main Logs

The CloverETL Server uses the log4j library for logging. The WAR file contains the default log4j configuration.

The log4j configuration file log4j.xml

is placed in

WEB-INF/classes

directory.

By default, log files are produced in the directory specified by system property "java.io.tmpdir" in the cloverlogs

subdirectory.

"java.io.tmpdir" usually contains common system temp dir i.e.

/tmp

. On Tomcat, it is usually

$TOMCAT_HOME/ temp

The default logging configuration (log4j.xml bundled in the clover.war) may be changed to another log4j configuration file using system property log4j.configuration

. If you override the configuration, only the properties from the new file are used.

The log4j.configuration

should contain the URL of the a new log4j configuration file. It's not just file system path, it must be URL, so it may look like this: log4j.configuration=file:/home/clover/config/log4j.xml

It is better to copy the original file and modify the copy, than to create a new one.

Please note, that "log4j.configuration" is not a CloverETL Server config property, but system property, thus it must be set on the JVM command line by -Dlog4j.configuration or in other way suitable for the application container.

Best possibility how to set a system property for each application container is described in the "Installation" chapter.

Since such a configuration overrides the default configuration, it may have influence over Graph run logs. So your own log config has to contain following fragment to preserve Graph run logs

</logger>

Another Useful Logging Settings

These system properties allow for logging of HTTP requests/responses to stdout:

Client side: com.sun.xml.ws.transport.http.client.HttpTransportPipe.dump=true

(for more information consult CloverETL Designer Users's Guide - chapter Integrating CloverETL Designer with

CloverETL Server )

Server side: com.sun.xml.ws.transport.http.HttpAdapter.dump=true

Chapter 11. Logging

Access Log in Apache Tomcat

If you need to log all requests processed by server, add the following code to

$CATALINA_HOME/conf/ server.xml

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"

prefix="localhost_access_log." suffix=".txt"

pattern="%h %l %u %t %D %r %s %b" />

The format defined above has following meaning

[IP address] [date-time] [processing duration in millis] [method] [URL] [protocol+version] [response code] [response size]

The log will look like the next line

172.17.30.243 - - [13/Nov/2014:12:53:03 +0000] 2 "POST /clover/sDispatcher/clusterNodeApi HTTP/1.1" 200 1435"

Graph Run Logs

Each graph or jobflow run has its own log file – for example, in the Server Console, section "Executions History".

By default, these log files are saved in the subdirectory cloverLogs/graph in the directory specified by

"java.io.tmpdir" system property.

It’s possible to specify a different location for these logs with the CloverETL "graph.logs_path" property. This property does not influence main Server logs.

Server Audit Logs

Server Audit Log logs operations called on ServerFacade and JDBC proxy interfaces.

Audit logging can be enabled by setting (adding) the value of CloverETL property logging.logger.server_audit.enabled

to true . In server GUI, you can change the property value in Configuration

→

Setup

→

Configuration File. Audit logging is enabled by default.

The name of output file is server-audit.log

. The file is in the same directory as main server log files.

Default log level is DEBUG, so all operations which may do any change or another important operations (e.g.

login or openJdbcConnection) are logged. To enable logging of all operations, change log level to TRACE in the log4j configuration.

Each logged operation is logged by two messages: entering method and exiting method (if the exception is raised, it's logged instead of output parameters)

• Entering method (marked as "inputParams"). All method's parameters (except for passwords) are printed.

• Exiting method (marked as "outputParams"). Method's return value is printed.

• Exception in method (marked as "EXCEPTION"). Exception's stacktrace is printed.

Message also contains:

• username, if the user is known

• client IP address, if it's known

• cluster node ID

• Interface name and the operation name

Values of transient and lazy initialized (in entity classes) fields and fields with binary content are not printed.

Part IV. Administration

Chapter 12. Temp Space Management

Many of the components available in the CloverETL Server require temporary files or directories in order to work correctly. Temp space is a physical location on the file system where these files or directories are created and maintained. CloverETL Server allows you to configure and manage temp spaces - you can specify their locations, see usage of the filesystem etc.

To access this administration section, you need

Temp Space Management permission (p. 110).

Overview

The overview of temp spaces defined in CloverETL Server is available under Configuration > Temp space

management > Overview

The overview panel displays list of temp spaces for each node in the cluster. These properties are displayed for each temp space:

• Root Path - location of the temp space with unresolved placeholders (see note below for placeholders)

• Resolved Path - location of the temp space with resolved placeholders (see note below for placeholders)

• Free Space - remaining space for the temp space

• Filesystem Size - all available space for the temp space (actual size of the filesystem where the temp space resides)

• Filesystem Usage - size of used space in percentage

• Available - the directory exists and is writable

• Status - current status of temp space, can be

Active

Suspended

Note

It is possible to use system properties and environment variables as placeholders. See

Using environment variables and system properties (p. 85).

Figure 12.1. Configured temp spaces overview - one default temp space on each cluster node

Chapter 12. Temp

Space Management

Management

Temp space management offers an interface to add, disable, enable and delete a temp space. It is accessible under

Configuration > Temp space management > Edit.

The screen is divided in two drop-down areas: Global Configuration and Per Node Configuration. The Global

configuration manages temp spaces of standalone server or in case of a server cluster temp spaces on all its nodes.

The Per Node Configuration allows to maintain temp spaces on each particular node.

Initialization (p. 84)

Adding Temp Space (p. 84)

Using environment variables and system properties (p. 85)

Disabling Temp Space (p. 86)

Enabling Temp Space (p. 87)

Removing Temp Space (p. 87)

Initialization

When CloverETL Server is starting the system checks temp space configuration: in case no temp space is configured a new default temp space is created in the directory where java.io.tmpdir

system property points.

The directory is named as follows:

•

${java.io.tmpdir}/clover_temp

in case of a standalone server

•

${java.io.tmpdir}/clover_temp_<node_id>

in case of server cluster

Adding Temp Space

In order to define new temp space enter its path into text field under last row in the table and click the Add link.

If the directory entered does not exist, it will be created.

Tip

The main point of adding additional temp spaces is to enable higher system throughput - therefore the paths entered should point to directories residing on different physical devices to achieve maximal

I/O performance.

Chapter 12. Temp

Space Management

Figure 12.2. Newly added global temp space.

Using environment variables and system properties

Environment variables and system properties can be used in the temp space path as a placeholder; they can be arbitrarily combined and resolved paths for each node may differ in accord with its configuration.

Note

The environment variables have higher priority than system properties of the same name. The path with variables are resolved after system has added new temp space and when the server is starting.

In case the variable value has been changed it is necessary to restart the server so that the change takes effect.

Examples:

• Given that an environment variable USERNAME has a value Filip . and is used as a placeholder in the path

C:\Users\${USERNAME}\tmp , the resolved path is C:\Users\Filip\tmp .

• Given that Java system property java.io.tmpdir

has a value

C:\Users\Filip\AppData\Local

\Temp

and the property is used as a placeholder in the path

${java.io.tmpdir}\temp_folder

, the resolved path is

C:\Users\Filip\AppData\Local\Temp\temp_folder

• Node node01

has been started with

-Dcustom.temporary.dir=C:\tmp_node01

parameter. Node node02

has been started with

-Dcustom.temporary.dir=C:\tmp_node02

parameter. The declared path is

${custom.temporary.dir}

. The resolved path is different for each node,

C:\tmp_node01

for node01

and

C:\tmp_node02

for node02

• When the declared path is

${java.io.tmpdir}\${USERNAME}\tmp_folder

, the resolved path is

\tmp\Filip\tmp_folder

Chapter 12. Temp

Space Management

Figure 12.3. Temp spaces using environment variables and system properties

Disabling Temp Space

To disable a temp space click on "Disable" link in the panel. Once the temp space has been disabled, no new temporary files will be created in it, but the files already created may be still used by running jobs. In case there are files left from previous or current job executions a notification is displayed.

Note

The system ensures that at least one enabled temp space is available.

Chapter 12. Temp

Space Management

Figure 12.4. Disable operation reports action performed

Enabling Temp Space

To enable a temp space click on "Enable" link in the panel. Enabled temp space is active, i.e. available for temporary files and directories creation.

Removing Temp Space

To remove a temp space click on "Remove" link in the panel. Only the disabled temp space may be removed. If there are any running jobs using the temp space, the system will not allow its removal. In case there are some files left in the temp space directory, it is possible to remove them in the notification panel. The available options are:

• Remove - remove temp space from a system, but keep its content

• Remove and delete - remove the temp space from a system and its content too

• Cancel - do not proceed with operation

Chapter 12. Temp

Space Management

Figure 12.5. Remove operation asks for confirmation in case there are data present in the temp space

Chapter 13. Secure Parameters

Transformation graphs in CloverETL Server environment allow you to define secure graph parameters. Secure graph parameters are regular graph parameters, either internal or external (in a *.prm file), but the values of the graph parameters are not stored in plain text on the file system - encrypted values are persisted instead. This allows you to use graph parameters to handle sensitive information, typically credentials such as passwords to databases.

Secure parameters are only available in CloverETL Server environment, including working with CloverETL

Server Projects in CloverETL Designer.

The encryption algorithm must be initialized with a master password. The master password has to be manually set after server installation in Configuration > Secure Parameters > Master password. Secure parameters cannot be used before the master password is set.

Figure 13.1. Master password initialization

After setting the master password secure parameters are fully available in Graph parameter editor in CloverETL

Designer. When setting value of a secure parameter, it will be automatically encrypted using the master password.

Secure parameters are automatically decrypted by server in graph runtime. A parameter value can also be encrypted in the CloverETL Server Console in the Configuration > Secure Parameters page - use the Encrypt text section.

Figure 13.2. Graph parameters tab with initialized master password

If you change the master password, the secure parameters encrypted using the old master password cannot be decrypted correctly anymore. In that case existing secure parameters need to be encrypted again with the new master password. That can be accomplished simply by setting their value (non-encrypted) again in the Graph

parameter editor. Similar master password inconsistency issue can occur if you move a transformation graph with some secure parameters to another server with a different master password. So it is highly recommended to use the identical master password for all your CloverETL Server installations.

Chapter 13. Secure Parameters

See documentation of secure parameters in CloverETL Designer manual for further details.

Secure parameters configuration

Encryption of secure parameters can be further customized via server configuration parameters.

Table 13.1. Secure parameters configuration parameters

Property name Default value

security.job_parameters.encryptor.algorithm

PBEWithMD5AndDES

Description

The algorithm to be used for encryption. This algorithm has to be supported by your JCE provider

(if you specify a custom one, or the default JVM provider if you don't). The name of algorithm should start with PBE prefix.

The list of available algorithms depends on your JCE provider, e.g. for the default SunJCE provider you can find them on http://docs.oracle.com/javase/6/docs/technotes/ guides/security/

SunProviders.html#SunJCEProvider or for the

Bouncy

Castle provider on http:// www.bouncycastle.org/specifications.html

(section Algorithms/PBE)).

security.job_parameters.encryptor.master_password_encryption.password

clover The password used to encrypt values persisted in the database table secure_param_passwd (the master password is persisted there).

security.job_parameters.encryptor.providerClassName

Empty string. The default

JVM provider is used (e.g.

for Oracle Java the SunJCE provider is used)

The name of the security provider to be asked for the encryption algorithm.

It must implement

java.security.Provider

interface. For example set to

org.bouncycastle.jce.provider.BouncyCastleProvider

for the Bouncy Castle JCE provider, see below.

Installing Bouncy Castle JCE provider

Algorithms provided by JVM could be too weak to satisfy an adequate security. Therefore it is recommended to install a third-party JCE provider. Following example demonstrates installation of one concrete provider, Bouncy

Castle JCE provider. Another provider would be installed similarly.

1. Download Bouncy Castle provider jar (e.g. bcprov-jdk15on-150.jar) from http://bouncycastle.org/ latest_releases.html

2. Add the jar to the classpath of your application container running CloverETL Server, e.g. to directory

WEB-

INF/lib

3. Set value of the security.job_parameters.encryptor.providerClassName attribute to

org.bouncycastle.jce.provider.BouncyCastleProvider in file WEB-INF/config.properties

4. Set value of the security.job_parameters.encryptor.algorithm attribute to the desired algorithm (e.g.

PBEWITHSHA256AND256BITAES-CBC-BC).

Example of configuration using Bouncy Castle:

Chapter 13. Secure Parameters

security.job_parameters.encryptor.algorithm=PBEWITHSHA256AND256BITAES-CBC-BC

security.job_parameters.encryptor.providerClassName=org.bouncycastle.jce.provider.BouncyCastleProvider

Chapter 14. Users and Groups

The CloverETL Server has a built-in security module that manages users and groups. User groups control access permissions to sandboxes and operations the users can perform on the Server, including authenticated calls to

Server API functions. A single user can belong to multiple groups.

LDAP or Active Directory can be configured with the Server to authenticate users and optionally assign their effective groups (and permissions) from a global directory.

You can manage users and user groups in Configuration/Users and Groups. Please note that you need a “List users” (“List groups” respectively) permission for that.

Chapter 14. Users and Groups

LDAP Authentication

Since 3.2 it's possible to configure CloverETL Server to use LDAP server for users authentication. So the credentials of users registered in LDAP may be used for authentication to any CloverETL Server interface (API or web console).

However authorization (access levels to sandboxes content and privileges for operations) is still handled by Clover security module. Each user, event though logged-in using LDAP authentication, must have his own "user" record

(with related groups) in the CloverETL security module. So there must be the user with the same username and domain set to "LDAP". Such record has to be created by a Server administrator before the the user can log in.

What does the CloverETL do to authenticate an LDAP user?

1. User specifies the LDAP credentials in a login form to the Server web console

2. CloverETL Server looks up a user record and checks whether has "LDAP" domain set

3. If the system is configured to use LDAP for authentication only, it attempts to connect to LDAP server using user's credentials. If it succeeds, the user is logged in.

4. In case the system is configured for user group synchronization the procedure is as follows:

5. CloverETL Server connects to the LDAP server and checks whether the user exists (it uses specified search to lookup in LDAP).

6. If the user exists in LDAP, CloverETL Server performs authentication.

7. If succeeded, CloverETL Server searches LDAP for user's groups.

8. Clover user is assigned to the Clover groups according to his current assignation to the LDAP groups.

9. User is logged-in.

Note

Switching domains:

• If a user was created as LDAP and then switched to clover domain, you have to set a password for him in Change password tab.

• If a user was created as clover and then switched to LDAP domain, he has a password in clover domain, but it is overridden by the LDAP password. After switching back to clover domain, the original password is re-used. It can be reset in the Change password tab if needed (e.g.

forgotten).

Configuration

By default CloverETL Server allows only its own internal mechanism for authentication. To enable authentication with LDAP, set the configuration property "security.authentication.allowed_domains" properly. It is a list of user domains that are used for authentication.

Currently there are 2 authentication mechanism implemented: "LDAP" and "clover" ("clover" is an identifier of

CloverETL internal authentication and may be changed by security.default_domain property, but only for whitelabelling purposes). To enable LDAP authentication, set value to "LDAP" (only LDAP) or "clover,LDAP". Users from both domain may login. It's recommended to allow both mechanisms together, until the LDAP is properly configured. So the admin user can still login to web GUI although the LDAP connection isn't properly configured.

You can use Setup to configure LDAP authentication. See the section called “ LDAP

” (p. 50) in Chapter 7,

Setup (p. 45).

Chapter 14. Users and Groups

Basic LDAP connection properties

# Implementation of context factory security.ldap.ctx_factory=com.sun.jndi.ldap.LdapCtxFactory

# URL of LDAP server security.ldap.url=ldap://hostname:port

# User DN pattern that will be used to create LDAP user DN from login name. security.ldap.user_dn_pattern=uid=${username},dc=company,dc=com

Depending on the LDAP server configuration the property security.ldap.user_dn_pattern

can be pattern for user's actual distinguished name in the LDAP directory, or just the login name - in such case just set the property to ${username} .

Configuration of user and group lookup

In order to be able to synchronize the Clover groups with those defined in LDAP directory, the security.ldap.user_dn_pattern

has to be left unspecified. There are additional properties required so that the server is able to search the LDAP directory.

# User DN of a user that has sufficient privileges to search LDAP for users and groups security.ldap.userDN=cn=Manager,dc=company,dc=com

# The password for user mentioned above.

security.ldap.password=

There are optional settings affecting how the LDAP directory is searched.

# Timeout for queries searching the directory.

security.ldap.timeout=5000

# Maximal number of records that the query can return.

security.ldap.records_limit=2000

# How LDAP referrals are processed, possible values are: 'follow', 'ignore' and 'throw'.

# The default depends on the context provider. security.ldap.referral=

Specified values work for this specific LDAP tree:

• dc=company,dc=com

• ou=groups

• cn=admins

(objectClass=groupOfNames,member=(uid=smith,dc=company,dc=com),member=(uid=jones,dc=company,dc=com))

• cn=developers (objectClass=groupOfNames,member=(uid=smith,dc=company,dc=com))

• cn=consultants (objectClass=groupOfNames,member=(uid=jones,dc=company,dc=com))

• ou=people

• uid=smith (fn=John,sn=Smith,[email protected])

• uid=jones (fn=Bob,sn=Jones,[email protected])

Following properties are necessary for lookup for the LDAP user by his username. (step [4] in the login process above)

# Base specifies the node of LDAP tree where the search starts security.ldap.user_search.base=dc=company,dc=eu

# Filter expression for searching the user by his username.

# Note, that this search query must return just one record.

# Placeholder ${username} will be replaced by username specified by the logging user.

Chapter 14. Users and Groups security.ldap.user_search.filter=(uid=${username})

# Scope specifies type of search in "base". There are three possible values: SUBTREE | ONELEVEL | OBJECT

# http://download.oracle.com/javase/6/docs/api/javax/naming/directory/SearchControls.html

security.ldap.user_search.scope=SUBTREE

Following properties are names of attributes from the search defined above. They are used for getting basic info about the LDAP user in case the user record has to be created/updated by Clover security module: (step [6] in the login process above) security.ldap.user_search.attribute.firstname=fn security.ldap.user_search.attribute.lastname=sn security.ldap.user_search.attribute.email=mail

# This property is related to the following step "searching for groups".

# Groups may be obtained from specified user's attribute, or found by filter (see next paragraph)

# Leave this property empty if the user doesn't have such attribute.

security.ldap.user_search.attribute.groups=memberOf

In the following step, clover tries to find groups which the user is assigned to. (step [4] in the login process above).

There are two ways how to get list of groups which the user is assigned to. The user-groups relation is specified on the "user" side. The user record has some attribute with list of groups. It's "memberOf" attribute usually. Or the relation is specified on the "group" side. The group record has an attribute with list of assigned users. It's

"member" attribute usually.

In case the relation is specified on users side, please specify property: security.ldap.user_search.attribute.groups=memberOf

Leave it empty otherwise.

In case the relation is specified on the groups side, set properties for searching: security.ldap.groups_search.base=dc=company,dc=com

# Placeholder ${userDN} will be replaced by user DN found by the search above

# If the filter is empty, searching will be skipped.

security.ldap.groups_search.filter=(&(objectClass=groupOfNames)(member=${userDN})) security.ldap.groups_search.scope=SUBTREE

Otherwise, please leave property security.ldap.groups_search.filter empty, so the search will be skipped.

Clover user record will be assigned to the clover groups according to the LDAP groups found by the search (or the attribute). (Groups synchronization is performed during each login)

# Value of the following attribute will be used for lookup for the Clover group by its code.

# So the user will be assigned to the Clover group with the same "code" security.ldap.groups_search.attribute.group_code=cn

Chapter 14. Users and Groups

Users

This section is intended to users management. It offers features in dependence on user's permissions. I.e., user may enter this section, but cannot modify anything. Or user may modify, but cannot create new users.

All possible features of users section:

• create new user

• modify basic data

• change password

• disable/enable user

• assign user to groups - Assignment to groups gives user proper permissions

Table 14.1. After default installation on an empty DB, admin user is created automatically

clover

User name Description

Clover user has admin permissions, thus default password "clover" should be changed after installation.

Figure 14.1. Web GUI - section "Users" under "Configuration"

Table 14.2. User attributes

Domain

Username

Password

Attribute Description

Domain which is the origin of the user. There are only two possible values currently: "clover" or "ldap".

Common user identifier. Must be unique, cannot contain spaces or special characters, just letters and numbers.

Case sensitive password. If user looses his password, the new one must be set. Password is stored in encrypted form for security reasons, so it cannot be retrieved from database and must be changed by the user who has proper permission for such operation.

First name

Last name

E-mail E-mail address which may be used by CloverETL administrator or by CloverETL server for automatic notifications. See

Send an

Email (p. 145) for details.

Chapter 14. Users and Groups

Edit user record

User with permission "Create user" or "Edit user" can use this form to set basic user parameters.

Figure 14.2. Web GUI - edit user

Change users Password

If user looses his password, the new one must be set. So user with permission "Change passwords" can use this form to do it.

Figure 14.3. Web GUI - change password

Group assignment

Assignment to groups gives user proper permissions. Only logged user with permission "Groups assignment" can access this form and specify groups which the user is assigned in. See

Groups (p. 99) for details about

permissions.

Chapter 14. Users and Groups

Figure 14.4. Web GUI - groups assignment

Disabling / enabling users

Since user record has various relations to the logs and history records, it can't be deleted. So it's disabled instead.

It basically means, that the record doesn't display in the list and the user can't login.

However disabled user may be enabled again. Please note, that disabled user is removed from its groups, so groups should be assigned properly after re-enabling.

Chapter 14. Users and Groups

Groups

Group is an abstract set of users, which gives assigned users some permissions. So it is not necessary to specify permission for each single user.

There are independent levels of permissions implemented in CloverETL Server

• permissions to Read/Write/eXecute in sandboxes - sandbox owner can specify different permissions for different groups. See

Sandbox Content Security and Permissions (p. 114) for details.

• permissions to perform some operation - user with operation permission "Permission assignment" may assign specific permission to existing groups.

• permissions to launch specific service - see

Launch Services (p. 216) for details.

Table 14.3. Default groups created during installation

admins all users

Group name Description

This group has operation permission "all" assigned, which means, that it has unlimited permission. Default user "clover" is assigned to this group, which makes him administrator.

Every single CloverETL user is assigned to this group by default.

It is possible to remove user from this group, but it is not a recommended approach. This group is useful for some permissions to sandbox or some operation, which you would like to make accessible for all users without exceptions.

Figure 14.5. Web GUI - section "Groups"

Users Assignment

Relation between users and groups is N:M. Thus in the same way, how groups are assignable to users, users are assignable to groups.

Chapter 14. Users and Groups

Figure 14.6. Web GUI - users assignment

Groups permissions

Groups permissions are structured as a tree, where permissions are inherited from the root to leafs. Thus if some permission (tree node) is enabled (blue dot), all permissions in sub tree are automatically enabled (white dot).

Permissions with red cross are disabled.

Thus for "admin" group just "all" permission is assigned, every single permission in sub tree is assigned automatically.

Figure 14.7. Tree of permissions

With no of the following privileges, user can: login to the server console, create server project (in Designer) from its own sandbox, create a file in its own existing sandbox, and run graphs.

• all

A user with this permission has all available permissions. Admin group has all permissions by default.

• Unlimited access to sandboxes

This permission allows user to perform operations on all sandboxes, even if the sandbox accessibility is not specified explicitly.

Unlimited access to sandboxes permission does not include the suspend sandbox permission (p. 107).

• Sandboxes

This permission allows user work with sandboxes. This permission contains all the permissions below. The user can perfom operations only on sandboxes owned by himself or on sandboxes with explicitly added access to him.

100

Chapter 14. Users and Groups

See Chapter 15, Server Side Job Files - Sandboxes (p. 111).

• List sandbox

In server web interface, this permission allows user to list her sandboxes and list sandboxes with read permission granted to the user's group.

In server web interface, this permission is necessary to create, edit, or delete sandboxes.

Within a sandbox with write access granted, user can edit or remove files and create or delete directories even without this permission.

• Create sandbox

This permission allows user to create a new sandbox.

If the sandbox is to be created in web interface, the user is required to have the

• Delete sandbox

This permission allows user to delete a sandbox.

If the sandbox is to be deleted in web interface, the user is required to have the

• Edit sandbox

This permission allows user to edit a sandbox.

If the sandbox is to be modified in web interface, the user is required to have the

• May delete files missing in uploaded ZIP

In Sandbox

→

Upload ZIP, this permission allows user to use a checkbox to delete files missing in the

ZIP to be uploaded. If the user does not have this permission, the checkbox to delete mission files in

ZIP is not displayed.

If the sandbox is to be uploaded from a ZIP file in server web interface, it is required to have the

See Chapter 22, Scheduling (p. 165).

• Scheduling

This permission allows user to manage schedules.

• List schedule

This permission allows user to list all schedules.

• List schedule limited

This permission allows user to list the enabled schedules.

• Create schedule

This permission allows user to create a new schedule.

101

Chapter 14. Users and Groups

The user needs the

list schedule limited permission (p. 101) to access the scheduling section to create

a new schedule.

• Delete schedule

This permission allows user to delete the schedule.

User needs

list schedule limited permission list schedule permission (p. 101) to access

the scheduling section to delete the schedule.

• Edit schedule

This permision allows user to edit the schedule.

User needs

list schedule limited permission list schedule permission (p. 101) to access

the scheduling section to edit the schedule.

• Event listeners

This permission allows user to manage event listeners.

See Chapter 24, Listeners (p. 176).

• List of Event Listeners

This permission allows user to list all event listeners.

• List of Jobflow Event Listeners unlimited

This permission allows user to list jobflow event listeners.

See Jobflow Event Listeners (p. 184)

• List of Jobflow Event Listeners limited

This permission allows user to list jobflow event listeners of sandboxes the user can read from.

• List of Graph Event Listeners unlimited

This permission allows user to list all graph event listeners.

See File Event Listeners (remote and local) (p. 194).

• List of Graph Event Listeners limited

This permission allows user to list graph event listeners from sandboxes the user can read from.

• List of File Event Listeners unlimited

This permission allows user to list all file event listeners.

• List of File Event Listeners limited

This permission allows user to list all file event listeners.

• List of JMS Event Listeners unlimited

This permission allows user to list all JMS listeners.

See JMS Messages Listeners (p. 187).

102

Chapter 14. Users and Groups

• List of JMS Event Listeners limited

This permission allows user to list all JMS listeners.

• List of Universal Event Listeners unlimited

This permission allows user to list all universal event listeners.

• List of Universal Event Listeners limited

This permission allows user to list all universal event listeners.

list of jobflow event listeners limited permission (p. 102).

• Create Event Listener

This permission allows user to create event listeners.

If the event listener is to be created in server web interface, the user needs to have permission to list the event listeners of the particular type.

• Create Jobflow Event Listener

This permission allows user to create a new Jobflow Event listener.

If the jobflow event listener is to be created in server web interface, the user needs to have the

See Jobflow Event Listeners (p. 184).

• Create Graph Event Listener

This permission allows user to create a graph event listener.

If the graph gvent listener is to be created in server web interface, the user needs to have the list of graph event listeners limited permission (p. 102).

list of file event listeners limited permission (p. 102).

• Create File Event Listener

This permission allows user to create a graph event listener.

If the file event listener is to be created in server web interface, the user needs to have the

See File Event Listeners (remote and local) (p. 194).

• Create JMS Listener

This permission allows user to create a JMS event listener.

If the JMS event listener is to be created in server web interface, the user needs to have the

list of JMS event listeners limited permission (p. 103).

See JMS Messages Listeners (p. 187).

• Create Universal Event Listener

103

Chapter 14. Users and Groups

This permission allows user to create a universal event listener.

If the universal event listener is to be created in server web interface, the user needs to have the

list of universal event listeners limited permission (p. 103).

list of jobflow event listeners limited permission (p. 102).

• Edit Event Listener

This permission allow user to edit an event listener.

If the event listener is to be created in server web interface, the user needs to have permission to list event listener of the particular type.

• Edit Jobflow Event Listener

This permission allows user to edit a jobflow event listener.

If the jobflow event listener is to be edited in server web interface, the user needs to have the

See Jobflow Event Listeners (p. 184).

• Edit Graph Event Listener

This permission allows user to edit a graph event listener.

If the graph event listener is to be edited in server web interface, the user needs to have the

list of graph event listeners limited permission (p. 102).

If the file event listener is to be edited in server web interface, the user needs to have the list of file event listeners limited permission (p. 102).

• Edit File Event Listener

This permission allows user to edit a file event listener.

See File Event Listeners (remote and local) (p. 194).

• Edit JMS Event Listener

This permission allows user to edit a JMS event listener.

If the JMS event listener is to be edited in server web interface, the user needs to have the

list of JMS event listeners limited permission (p. 103).

• Edit Universal Event Listener

This permission allows user to edit a universal event listener.

If the universal event listener is to be edited in server web interface, user needs to have permission

list of universal event listeners limited permission (p. 103).

delete graph event listener permission (p. 105) to delete a jobflow event

• Delete Event Listener

This permission allows user to delete event listeners.

104

Chapter 14. Users and Groups

• Delete Jobflow Event Listener

This permisison allows user to delete a jobflow event listener.

User needs to have the

listener.

It the jobflow event listener is to be deleted in server web interface, the user needs to have the

list of jobflow event listeners limited permission (p. 102)

• Delete Graph Event Listener

This permission allows user to delete a graph event listener.

If the graph event listener is to be deleted in server web interface, the user needs to have the list of graph event listeners limited permission (p. 102).

The user needs to have the delete graph event listener permission (p. 105) to delete a file event listener.

• Delete File Event Listener

This permission allows user to delete a file event listener.

If the file event listener is to be deleted in server web interface, the user needs to have the

list of file event listeners limited permission (p. 102).

See File Event Listeners (remote and local) (p. 194).

• Delete JMS Event Listener

This permission allows user to delete a JMS Event Listener.

The user needs to have the

delete graph event listener permission (p. 105) to delete a JMS event

listener.

If the graph event listener is to be deleted in server web interface, the user needs to have the list of JMS event listeners limited permission (p. 103).

• Delete Universal Event Listener

This permission allows user to delete a universal event listener.

The user needs to have the

delete graph event listener permission (p. 105) to delete universal event

listener.

If the universal event listener is to be deleted in server web interface, the user needs to have the

list of universal event listeners limited permission (p. 103).

See Chapter 21, Manual Task Execution (p. 164).

• Manual task Execution

This permission allows user to manually execute a task (send an email, execute a script, etc.) with an immediate effect.

• Unlimited access to execution history

105

Chapter 14. Users and Groups

This permission allows user to perform the same operations as

unlimited access to execution history list permission (p. 106).

• Unlimited access to execution history list

This permission allows user to view execution history of all jobs.

• Limited access to execution history list

This permission allows user to view execution history of jobs from sandboxes the user can read from.

In Designer, this permission is required to be able to view Execution log in Designer's console and execution history in Execution tab.

• Launch Services

This permission allows user to list, create, edit, and delete launch services.

See Launch Services (p. 216).

• List Launch Services unlimited

This permission allows user to list all launch services.

• List Launch Services Limited

This permission allows user to list launch services from sandboxes the user can read from.

• Create Launch service

This permission allows user to create a new launch service.

User has to have the

create graph event listener permission (p. 103) to bind the launch service with a

graph.

If the launch service is to be created in server web interface, the user has to have the

list launch services limited permission (p. 106) (or the list launch services unlimited permission

section with launch services.

• Delete Launch Service

This permission allows user to delete a launch service.

User has to have delete graph event listener permission (p. 105) to delete a launch service.

If the launch service is to be deleted in server web interface, the user needs to have the list launch services limited permission (p. 106) to access the section with launch services.

• Edit Launch Service

This permission allows user to edit a launch services.

User has to have edit graph event listener (p. 104) to edit the launch service.

If the launch service is to be edited in server web interface, the user needs to have the

list launch services limited permission (p. 106) to choose the launch service in the server interface.

• Tasks history

This permission allows user to access Tasks history section.

See Chapter 20, Tasks (p. 144).

106

Chapter 14. Users and Groups

• Monitoring

Monitoring permission grants user all its subpermissions.

• Monitoring section

This permission allows user to access the monitoring section.

See Chapter 16, CloverETL Server Monitoring (p. 124).

• Suspend

This permission allows user to suspend the server, a cluster node, or a sandbox.

The user needs to have the monitoring section permission (p. 107) to access the Monitoring section.

• Suspend server

This permission allows user to suspend or resume the server.

The user needs to have the monitoring section permission (p. 107) to access the monitoring section.

• Suspend cluster node

This permission allows user to suspend or resume a cluster node.

The user needs to have the monitoring section permission (p. 107) to access the monitoring section.

• Suspend sandbox

This permission allows user to suspend a sandbox. The user needs to have

list sandbox permission (p. 101) to view the sandboxes to suspend them.

• Reset caches

Deprecated.

• Running jobs unlimited

If the graph is to be run from server web interface, the user needs to have the

list sandbox permission (p. 101) to list the graphs.

• Running jobs limited

If the graph is to be run from server web interface, the user needs to have the

list sandbox permission (p. 101) to list the graphs.

• Configuration

This permission allows user to access the configuration section.

• Users

This permission allow user to access the Users section and configure user accounts.

• List user

This permission allows user to list users and access to the Users administation section (Configuration

→

Users)

107

Chapter 14. Users and Groups

• Change passwords

This permission allows user to change his password and to change password of another user.

To see list of users, the user needs the

list user permission (p. 107).

• Edit user

This permission allows user to change group assignment.

To see the list of users, the user needs to have the list user permission (p. 107).

• Edit own profile and password

This permisison allows user to change his profile (first name, last name, email, and password).

The user can access her profile in main web onsole view under username, in upper right corner of the page.

• Delete user

This permission allows user to disable a user.

The user needs to have the list user permission (p. 107) to list available users.

• Create user

This permission allows user to create a new user.

If the user is to be created in server web interface, the creating user needs to have the

list user permission (p. 107) to list users to access this option.

• Groups assignement

This permission allows user to assign users to groups.

The user needs to have the

edit user permission (p. 108) to sucessfully finish the assigment of users

to groups.

If the user is to be created in server web interface, the creating user needs to have the

list user permission (p. 107) to list users to access this option.

• Groups

This permission allows user to manage groups: user can list groups, create groups, delete groups, edit the group, assign users to the group, and change permissions of the group.

• List groups

This permission allows user to list groups. This permission is necessary for use of other options from the

Groups group.

• Create group

This permission allows user to create a new user group.

If the user group is to be created in server web interface, the user needs to have the

list groups permission (p. 108) to view a list of groups and to access this option.

• Delete group

108

Chapter 14. Users and Groups

This permission allows user to delete a user group.

Only empty groups can be deleted. You need to have the

list groups permission (p. 108) to view list

of groups and to access this option.

• Edit group

This permission allow user to edit user groups.

This permission does not include User assignment and Permission assignment.

If the user group is to be edited from server web interface, the user needs to have the

list groups permission (p. 108).

• Users assignment

This permission allows user to assign users to groups.

The user needs

Edit group permission (p. 109) to commit the changes in the assignment.

If the assignment is to be edited in server web interface, the user needs to have the

list groups permission (p. 108) to list the groups.

• Permission assignment

This permission allows user to configure group Permissions.

The user needs have the

Edit group permission (p. 109) to commit the changes.

If the permissions are to be edited in server web interface, the user needs to have the

list groups permission (p. 108) to list the groups.

• Secure parameters administration

• Secure params

This permission allows user to change the value of a secure parameter.

The user can use secure parameters in graphs even without this permission.

• CloverETL/System info sections

This permission allows user to view System Info and CloverETL Info sections.

• CloverETL Server properties

This permission allows user to view Server Properties tab and Data Profiler properties tab in CloverETL

Info section.

The user needs to have the

CloverETL/System info sections permission (p. 109) to access CloverETL

Info section.

• Reload license

This permission allows user to reload and view the server license.

The user needs to have the

CloverETL/System info sections permission (p. 109) to access the

Configuration section.

• Upload license

109

Chapter 14. Users and Groups

This permission allows user to update the server license.

The user needs to have the

CloverETL/System info sections permission (p. 109) to access the

Configuration section.

See Installation of CloverETL Server License (p. 28).

• Server Configuration Management

This permission allows user to import and export the server configuration.

See Chapter 17, Server Configuration Migration (p. 132).

• Export Server Configuration

This permission allows user to export the server configuration.

See Server Configuration Export (p. 133).

• Import Server Configuration

This permission allows user to import the server configuration.

See Server Configuration Import (p. 134).

• Temp Space Management

This permission allows user to access Temp Space Management section.

See Chapter 12, Temp Space Management (p. 83).

• Server Setup

This permission allows user to access the server setup.

See Chapter 7, Setup (p. 45).

• Heap Memory Dump

This permission allows user to create a Thread dump and a Heap Memory Dump.

See Chapter 18, Diagnostics (p. 138).

• Groovy Code API

This permission allows user to run groovy scripts.

See Groovy Code API (p. 223).

• Open Profiler Reporting Console

This permission allows user to login to the Profiler reporting console.

The permission is necessary to view the results of Clover Profiling Jobs in Designer.

Even without this permission, a user can create and run .cpj jobs from Designer.

110

Chapter 15. Server Side Job Files - Sandboxes

A sandbox is a place where you store all your project’s transformation graph files, jobflows, data, and other resources. It’s a server side analogy to a Designer project. The Server adds additional features to sandboxes, like user permissions management and global per-sandbox configuration options.

The Server and the Designer are integrated so that you are able to connect to a Server sandbox using a “Server

Project” in your Designer workspace. Such a project works like a remote file system – all data is stored on the

Server and accessed remotely. Nonetheless, you can do everything with Server Projects the same way as with local projects – copy and paste files, create, edit, and debug graphs, etcetera. See the CloverETL Designer manual for details on configuring a connection to the Server.

Technically, a sandbox is a dedicated directory on the Server host file system and its contents are managed by the Server. Advanced types of sandboxes, like “partitioned sandbox” have multiple locations to allow distributed

parallel processing (more about that in Chapter 29, Clustering Features (p. 229)). A sandbox cannot contain

another sandbox within – it’s a single root path for a project.

It's recommended to put all sandboxes in a folder outside the CloverETL Server installation (by default the sandboxes would be stored in the ${user.data.home}/CloverETL/sandboxes, where the "user.data.home" is automatically detected user home directory). However, each sandbox can be located on the file system independently of the others if needed. The containing folder and all its contents must have read/write permission for the user under which the CloverETL Server/application server is running.

Figure 15.1. Sandboxes Section in CloverETL Server Web GUI

Each sandbox in non-cluster environment is defined by following attributes:

111

Chapter 15. Server Side

Job Files - Sandboxes

Table 15.1. Sandbox attributes

Sandbox ID

Sandbox

Sandbox root path

Owner

A unique "name" of the sandbox. It is used in server APIs to identify sandbox. It must meet common rules for identifiers. It is specified by user in during sandbox creation and it can be modified later. Note: modifying is not recommended, because it may be already used by some

APIs clients.

Sandbox name used just for display. It is specified by user in during sandbox creation and it can be modified later.

Absolute server side file system path to sandbox root. It is specified by user during sandbox creation and it can be modified later. Instead of the absolute path, it's recommended to use ${sandboxes.home} placeholder, which may be configurable in the CloverETL Server configuration. So e.g. for the sandbox with ID "dataReports" the specified value of the

"root path" would be "${sandboxes.home}/dataReports". Default value of "sandboxes.home" config property is "${user.data.home}/CloverETL/sandboxes" where the "user.data.home" is configuration property specifying home directory of the user running JVM process - it's OS dependent). Thus on the unix-like OS, the fully resolved sandbox root path may be: "/home/

clover/CloverETL/sandboxes/dataReports". See Chapter 29, Clustering Features

for details about sandboxes root path in cluster environment.

It is set automatically during sandbox creation. It may be modified later.

112

Chapter 15. Server Side

Job Files - Sandboxes

Referencing Files from the ETL Graph or Jobflow

In some components you can specify file URL attribute as a reference to some resource on the file system. Also external metadata, lookup or DB connection definition is specified as reference to some file on the filesystem.

With CloverETL Server there are more ways how to specify this relation.

• Relative path

All relative paths in your graphs are considered as relative paths to the root of the same sandbox which contains job file (ETL graph or Jobflow).

• sandbox:// URLs

Sandbox URL allows user to reference the resource from different sandboxes with standalone CloverETL Server or the cluster. In cluster environment, CloverETL Server transparently manages remote streaming if the resource is accessible only on some specific cluster node.

See Using a Sandbox Resource as a Component Data Source (p. 234) for details about the sandbox URLs.

113

Chapter 15. Server Side

Job Files - Sandboxes

Sandbox Content Security and Permissions

Each sandbox has its owner which is set during sandbox creation. This user has unlimited privileges to this sandbox as well as administrators. Another users may have access according to sandbox settings.

Figure 15.2. Sandbox Permissions in CloverETL Server Web GUI

Permissions to a specific sandbox are modifiable in Permissions tab in sandbox detail. In this tab, selected user groups may be allowed to perform particular operations.

There are 3 types of operations:

Table 15.2. Sandbox permissions

Read

Write

Execute

Profiler Read

Profiler Admin

Users can see this sandbox in their sandboxes list.

Users can modify files in the sandbox through CS APIs.

Users can execute jobs in this sandbox. Note: jobs executed by "graph event listener" and

similar features is actually executed by the same user as job which is source of event. See details in "graph event listener". Job executed by schedule trigger is actually executed by

the schedule owner. See details in Chapter 22,

Scheduling

(p. 165). If the job needs any

files from the sandbox (e.g. metadata), user also must have read permission, otherwise the execution fails.

User can view results of profiler jobs executed from the sandbox.

User can administer results of profiler jobs executed from the sandbox.

Please note that, these permissions modify access to the content of specific sandboxes. In additions, it's possible to configure permissions to perform operations with sandbox configuration. e.g. create sandbox, edit sandbox, delete

sandbox, etc. Please see Chapter 14, Users and Groups (p. 92) for details.

114

Chapter 15. Server Side

Job Files - Sandboxes

Sandbox Content

Chapter 15. Server Side

Job Files - Sandboxes

Job Config Properties

Each ETL graph or Jobflow may have set of config properties, which are applied during the execution. Properties are editable in web GUI section "sandboxes". Select job file and go to tab "Config properties".

The same config properties are editable even for each sandbox. Values specified for sandbox are applied for each job in the sandbox, but with lower priority then config properties specified for the job.

If neither sandbox nor job have config properties specified, defaults from main server configuration are applied. Global config properties related to Job config properties have prefix "executor.". E.g. server property

"executor.classpath" is default for Job config property "classpath". (See Part III, “ Configuration ” (p. 42) for details)

In addition, it is possible to specify additional job parameters, which can be used as placeholders in job XML.

Please keep in mind, that these placeholders are resolved during loading and parsing of XML file, thus such job couldn't be pooled.

If you use a relative path, the path is relative to

${SANDBOX_ROOT}

In path definition, you can use system properties - e.g.

${java.io.tmpdir}

- and some of server config properties:

${sandboxes.home}

${sandboxes.home.partitioned}

and

${sandboxes.home.local}

Table 15.4. Job config parameters

Property name

tracking_interval enqueue_executions log_level max_graph_instance_age classpath

2000 false

INFO

Default value

max_running_concurrently unlimited

Description

Interval in ms for sampling nodes status in running transformation.

Max number of concurrently running instances of this transformation. In cluster environment, the limit is per node.

Boolean value. If it is true, executions above max_running_concurrently are enqueued, if it is false executions above max_running_concurrently fail.

Log4j log level for this graph executions. (ALL

| FATAL) For lower levels (ALL, TRACE or

DEBUG), also root logger level must be set to lower level. Root logger log level is INFO by default, thus transformation run log does not contain more detail messages then INFO event if job config

parameter "log_level" is set properly. See Chapter 11,

Logging (p. 80) for details about log4j configuration.

A time interval in ms which specifies how long may a transformation instance last in server's cache.

0 means that the transformation is initialized and released for each execution. The transformation cannot be stored in the pool and reused in some cases (a transformation uses placeholders using dynamically specified parameters)

List of paths or jar files which contain external classes used in the job file (transformations, generators, JMS processors). All specified resources will be added to runtime classpath of the transformation job. All

Clover Engine libraries and libraries on applicationserver's classpath are automatically on the classpath. Separator is specified by Engine property

119

Property name

compile_classpath classloader_caching skip_check_config password verbose_mode use_jmx debug_mode

Chapter 15. Server Side

Job Files - Sandboxes

Default value Description

"DEFAULT_PATH_SEPARATOR_REGEX".

Directory path must always end with a slash character

"/", otherwise ClassLoader doesn't recognize it's a directory. Server always automatically adds "trans" subdirectory of job's sandbox, so It doesn't have to be added explicitly.

List of paths or jar files which contain external classes used in the job file (transformations, generators, JMS processors) and related libraries for their compilation.

Please note, that libraries on applicationserver's classpath aren't included automatically.

Separator is specified by Engine property

"DEFAULT_PATH_SEPARATOR_REGEX". The directory path must always end with a slash character

"/", otherwise ClassLoader doesn't recognize it's a directory. Server always automatically adds

"SANDBOX_ROOT/trans/" directory and all JARs in "SANDBOX_ROOT/lib/" directory, so they don't have to be added explicitly.

false Clover creates new classloaders whenever is necessary to load a class in runtime. For example,

Reformat component with a Java transformation has to create a new classloader to load the class.

It is worth noting that classloaders for JDBC drivers are not re-created. Classloader cache is used to avoid PermGen out of memory errors

(some JDBC drivers automatically register itself to

DriverManager, which can cause the classloader cannot be released by garbage collector). This behaviour can be inconvenient for example if you want to share POJO between components. For example, a Reformat component creates an object

(from a jar file on runtime classpath) and stores it into a dictionary. Another Reformat component get the object from the dictionary and tries to cast the object to expected class. ClassCastException is thrown due different classloaders used in the

Reformat components. Using this flag you can force

CloverServer to re-use classloader when possible.

default value is taken from engine property

Switch which specifies whether check config must be performed before transformation execution.

true true false

This property is deprecated. Password for decoding of encoded DB connection passwords.

If true, more descriptive logs of job runs are generated.

If true, job executor registers jmx mBean of running transformation.

If true, edges with debug enabled will store data into files in a debug directory.

Without explicit setting, running of a graph from

Designer with server integration would set the debug_mode to true. On the other hand, running of a

120

Chapter 15. Server Side

Job Files - Sandboxes delete_obsolete_temp_files false use_local_context_url jobflow_token_tracking locale

Property name

time_zone

Default value Description

graph from the server console sets the debug_mode to false.

If true, system will remove temporary files produced during previous finished runs of the respective job.

false true

DEFAULT_LOCALE engine property

DEFAULT_TIME_ZONE engine property

This property is useful together with enabled debug mode ensuring that obsolete debug files from previous runs of a job are removed from temp space. This property is set to "true" by default when executing job using designer-server integration.

If true, the context URL of a running job will be a local "file:" URL. Otherwise, a "sandbox:" URL will be used.

If false, token tracking in jobflow executions will be disabled.

Can be used to override the DEFAULT_LOCALE engine property.

Can be used to override

DEFAULT_TIME_ZONE engine property.

the

Figure 15.9. Job config properties

121

Chapter 15. Server Side

Job Files - Sandboxes

WebDAV Access to Sandboxes

Since 3.1

WebDAV API allows you to access and manage sandbox content using a standard WebDAV specification.

Specifically, it allows for:

• Browsing a directory structure

• Editing files

• Removing files/folders

• Renaming files/folders

• Creating files/folders

• Copying files

• Moving files

The WebDAV interface is accessible from the URL: "http://[host]:[port]/clover/webdav".

Note: Although common browsers will open this URL, most of them are not rich WebDAV clients. Thus, you will only see a list of items, but you cannot browse the directory structure.

WebDAV Clients

There are many WebDAV clients for various operating systems, some OS support WebDAV natively.

Linux like OS

Great WebDAV client working on Linux systems is Konqueror. Please use different protocol in the URL: webdav://[host]:[port]/clover/webdav

Another WebDAV client is Nautilus. Use different protocol in the URL dav://[host]:[port]/clover/ webdav

MS windows

Last distributions of MS Windows (Win XP and later) have native support for WebDAV. Unfortunately, it is more or less unreliable, so it is recommended to use some free or commercial WebDAV client.

• The best WebDAV client we've tested is BitKinex: http://www.bitkinex.com/webdavclient

• Another option is to use Total Commander (http://www.ghisler.com/index.htm) with WebDAV plugin: http:// www.ghisler.com/plugins.htm#filesys

Mac OS

Mac OS supports WebDAV natively and in this case it should be without any problems. You can use "finder" application, select "Connect to the server ..." menu item and use URL with HTTP protocol: "http://[host]:[port]/ clover/webdav".

WebDAV Authentication/Authorization

CloverETL Server WebDAV API uses the HTTP Basic Authentication by default. However it may be reconfigured

to use HTTP Digest Authentication. Please see Part III, “ Configuration ” (p. 42) for details.

Digest Authentication may be useful, since some WebDAV clients can't work with HTTP Basic Authentication, only with Digest Authentication.

122

Chapter 15. Server Side

Job Files - Sandboxes

HTTP Digest Authentication is feature added to the version 3.1. If you upgraded your older CloverETL Server distribution, users created before the upgrade cannot use the HTTP Digest Authentication until they reset their passwords. So when they reset their passwords (or the admin does it for them), they can use Digest Authentication as well as new users.

123

Chapter 16. CloverETL Server Monitoring

Monitoring section in the server Web GUI displays useful information about current performance of the standalone

CloverETL Server or all cluster nodes if the clustering is enabled.

Monitoring section of the standalone server has slightly different design from cluster environment. In case of standalone server, the server-view is the same as node detail in cluster environment.

The section is refreshed each 15 seconds so the displayed data is up-to-date. Page can be also anytime refreshed manually by the "Refresh" button

Standalone Server Detail

Standalone server detail view displays info collected from the standalone server. The info is grouped in several panels. The following ones are displayed by default.

• Performance

• Resource utilization

• 10 longest-running jobs

• System

• Status history

You can display the hidden actions with Actions button: choose Actions

→

Show details.

Figure 16.1. Standalone server detail

124

Quartz panel lists scheduled actions: their name, description, start time, end time, time of previous event, time of next event and expected final event.

Figure 16.13. Quartz

128

Chapter 16. CloverETL

Server Monitoring

Cluster Overview

Cluster overview displays info collected from all cluster nodes. The info is grouped in several panels:

• List of nodes with a toolbar - allows manipulation with selected nodes

• Status history - Displays last 10 status changes for all cluster nodes

• Node detail - Displays several basic performance attributes for selected nodes. It's visible on the right side only when activated by button on the toolbar.

• Running jobs - It's displayed only when there are running jobs.

Figure 16.14. Cluster overview

129

Chapter 16. CloverETL

Server Monitoring

Node Detail

Node Detail is similar to the "Standalone server detail" mentioned above, however it displays detail info about node selected in the tree on the left.

Figure 16.15. Node detail

130

Chapter 16. CloverETL

Server Monitoring

Server Logs

Server Logs tab allows user to investigate log messages logged on other cluster nodes. Since the log messages are collected in memory, the maximum number of collected messages is relatively low by default, however it's customisable.

There are different "Log types":

• COMMON - Ordinary server logs as stored in log files.

• CLUSTER - Only cluster - related messages are visible in this log

• LAUNCH_SERVICES - Only requests for launch services

• AUDIT - Detail logging of operations called on the CloverETL Server core. Since the full logging may affect

server performance, it's disabled by default. See Server Audit Logs (p. 81) for details

• USER_ACTION - Contains some of user operations, e.g. login, logout, job execution

Figure 16.16. Server Logs

131

Chapter 17. Server Configuration Migration

CloverETL Server provides means to migrate its configuration (e.g. event listeners, schedules etc.) or parts of the configuration between separate instances of the server. A typical use case is deployment from test environment to production - this involves not only deployment of CloverETL graphs, but also copying parts of configuration such as file event listeners etc.

Configuration migration is performed in 2 steps - export of the configuration from the source server, followed by import of the configuration at the destination server. After exporting, the configuration is stored as an XML file. The file can be modified manually before import, for example to migrate only parts of the configuration.

Additionally, the configuration file can be stored in a versioning system (such as Subversion or Git) for versioning of the CloverETL Server configuration.

It is recommended to perform import of configuration on a suspended CloverETL Server and to plan for maintenance. Additionally, it is recommended to backup the CloverETL Server configuration database before the import.

The following items are parts of the Server Configuration and can be migrated between servers:

• Users & Groups (see Chapter 14, Users and Groups (p. 92))

• Sandboxes (see Chapter 15, Server Side Job Files - Sandboxes (p. 111))

• Job Parameters (see Chapter 19, Graph/Jobflow Parameters (p. 140))

• Schedules (see Chapter 22, Scheduling (p. 165))

• Event Listeners (see

Graph Event Listeners (p. 177),

Jobflow Event Listeners (p. 184),

JMS Messages

Listeners (p. 187),

File Event Listeners (remote and local) (p. 194))

• Launch Services (see Launch Services (p. 216))

• Temp Spaces (see Chapter 12, Temp Space Management (p. 83))

Permissions for Configuration Migration

Whether a user is entitled to perform configuration migration is determined by having Server Configuration

Management permission; this permission has two sub-permissions: Export Server Configuration and Import Server

Configuration (see

Groups permissions (p. 100) for further information on permissions). These permissions are

of higher priority than permissions related to a particular migrated item type - so even if the user does not have a permission e.g. to list server's schedules, with Export Server Configuration he will be allowed to export all of defined schedules. The same is true for adding and changing items with the Import Server Configuration permission.

See Server Configuration permission (p. 110).

132

Chapter 17. Server

Configuration Migration

Server Configuration Export

Export of a server configuration is performed from the Server Console - the screen for export can be found in

section Configuration > Export. You can choose which items will be exported (see Figure 17.1 (p. 133)).

After clicking on the Export Configuration an XML file will be offered for download. The name of the XML file reflects time when the configuration was exported.

In case user manually edits the exported XML file, you shell ensure that the file has a valid content. This can be done by validation against XSD schema. The schema for a configuration XML document can be found at http://[host]:[port]/[contextPath]/schemas/clover-server-config.xsd

The XML file contains selected items of the CloverETL server instance. The file can by modified before the import to another server instance - for example to import schedules only.

Figure 17.1. Server Configuration Export screen

133

Chapter 17. Server

Configuration Migration

Server Configuration Import

Import of CloverETL Server configuration merges the configuration exported from another server into the running server instance where the import was initiated. The configuration to be imported is loaded from an XML file

created by export, see Server Configuration Export (p. 133). Import of server configuration is performed from the

Server Console - the screen for import can be found in Configuration > Import section.

Figure 17.2. Server Configuration Import screen

The XML configuration file defines configuration items to be imported into the destination server. The items are matched against current configuration of the destination server. Depending on result of the matching, the items from the XML configuration are either added to the destination server or will update existing item with properties defined in the XML file. Matching of items is based on a key that depends on the item type:

• users - user code

• user groups - group code

• sandboxes - sandbox code

• job parameters - triplet (job parameter name, sandbox code, job file)

• event listeners - event listener name

• schedule - schedule description

• launch service - triplet (service name, server user, service user group)

• temp spaces - pair (temp space node ID, temp space path)

Configuration Import Process

Uploading Configuration

The first step in the configuration import is to upload the XML file to the CloverETL server. After clicking on CloverETL Configuration File button a window is opened where user can select an XML file with the configuration to import. The file is uploaded automatically after the dialog is closed. Once upload is finished the name of the uploaded file is shown in the toolbar along with CloverETL Configuration File button. In case reading of configuration from XML has finished without error, additional controls are displayed in the toolbar:

• Preview Import button to perform "dry run" of the configuration import

• Commit Import button to perform actual import of configuration to server's database

• Import Options section to further configure import process:

• New only option specifies that only new items will be imported leaving existing items on server untouched

• Import Items section to select what item types will be imported to the server

134

Chapter 17. Server

Configuration Migration

Figure 17.3. Server Configuration uploaded

Note

When transfering configuration from one server instance to another, it is important that these instances are of compatible, preferably the same, version. The user is notified when the source and target differ at at least minor version number (e.g. 3.4.1 and 3.5.0). It is also recommended not to transfer configuration between clustered and non-clustered server instances.

Verifying Configuration

Once the configuration is uploaded, the system executes "dry run" of the configuration import automatically. The

dry run performs configuration import, but no data is actually written do the server's database. The outcome of the operation is Import Log with messages related either to the configuration as a whole or to particular imported

items (see Figure 17.4 (p. 136)). There is also another view of Imported Items to list all added/updated items

grouped into tables according to their types. Each item has an icon indicating result of the item import operation:

• - the item has been added as a new one

• - the item has been updated

• - the item has been updated, but none of its properties has changed

• - the item has been removed

For the updated items, the state of the item before update is shown in the lower of the rows with less contrast text color, the new values of item's properties are shown in upper of the rows with regular text color. Actual changes are highlighted by background color change on respective property and also on both rows. The Imported Items view can be filtered using radio buttons above it:

• Changes only button will display only items that have been either added or actually changed by update

• All updates button will display all of imported items, event those identical to already present ones

Example 17.1. Example of simple configuration defining one new server user.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<username>johnsmith</username>

<lastName>Smith</lastName>

<email>[email protected]</email>

<domain>clover</domain>

<groupCode>all_users</groupCode>

135

Chapter 17. Server

Configuration Migration

<groupCode>job_managers</groupCode>

</userGroups>

</user>

</usersList>

</cloverServerConfiguration>

Figure 17.4. Outcome of the import preview for configuration from Example 17.1 (p. 135)

The Summary in the Import Log says whether the dry run was successful. Should there by any problems with

items imported, the item is displayed along with the cause of the error (see Figure 17.4 (p. 136)) .

136

Chapter 17. Server

Configuration Migration

Figure 17.5. Outcome of import preview for configuration after fixing by removal of broken group reference.

User is expected follow the advices displayed in the Import Log and edit the XML until import preview has finished without errors.

Committing Import

Once the import preview has finished without problems, one can proceed with actual configuration import. This is performed by clicking on the Commit Import button. After confirmation, Import Log will display outcome of the operation.

It is possible that some items will not be initialized properly after they have been imported (e.g. their initialization requires presence of a cluster node that went down in the meantime). User is notified about these problems in Import Log with link to the problematic item. One should check such items in appropriate section of the

CloverETL Server console and change their settings to fix the issue or remove them.

137

Chapter 18. Diagnostics

CloverETL Server allows you to create a thread dump or a heap dump. The thread and heap dumps are useful for investigation of performance and memory issues.

In server GUI, go to Configuration

→

System Info

→

Diagnostics.

Heap Dump

Heap Dump is content of a JVM process memory stored in a binary file.

To download the Heap Dump, click Download button under Heap Dump section. The download of Heap Dump may take some time.

The Dump live objects only checkbox allows you to avoid dumping of objects awaiting garbage collection

You can use jvisualvm or jhat to view and analyze the heap dump.

Important

Heap Dump does not work on WebSphere. On WebSphere, you can create heap dumps using administration console. See IBM Knowledge Center for the instructions.

Thread Dump

Thread Dump is a list of exisitng JVM threads with their callstacks and held locking objects (if supported). It can be viewed in a text editor.

To download the thread dump, click Download button under Thread Dump section.

Required Permissions

To create Thread Dump or Heap Dump, following permissions are required:

• Configuration →

Heap Memory Dump

138

Part V. Using Graphs

Chapter 19. Graph/Jobflow Parameters

The CloverETL Server passes a set of parameters to each graph or jobflow execution.

Keep in mind that

${paramName}

placeholders (parameters) are resolved only during the initialization (loading of XML definition file), so if you need the parameters to be resolved for each job execution, you cannot set the job to be pooled. However, current parameter values are always accessible by inline Java code like this:

String runId = getGraph().getGraphProperties().getProperty("RUN_ID");

Properties may be added or replaced like this: getGraph().getGraphProperties().setProperty("new_property", value );

This is set of parameters which are always set by CloverETL Server:

Table 19.1. Defaults for graph execution configuration - see section Graph config properties for details

key

SANDBOX_CODE

JOB_FILE

SANDBOX_ROOT

RUN_ID

PARENT_RUN_ID

ROOT_RUN_ID

CLOVER_USERNAME

NODE_ID

description

An identifier of a sandbox which contains executed graph.

A path to the file (graph, subgraph, jobflow). The path relative to sandbox root path.

An absolute path sandbox root.

ID of the graph execution. In standalone mode or in cluster mode, it is always unique. It may be lower then

0 value, if the run record isn't persistent. See

Launch

Services (p. 216) for details.

Run ID of the graph execution which is a parent to the current one. Useful when the execution is subgraph, child-job of some jobflow or worker for distributed transformation in cluster. When the execution doesn't have a parent, the PARENT_RUN_ID is the same as

RUN_ID.

Run ID of the graph execution which is root execution to the current one (the one which doesn't have parent).

Useful when the execution is a subgraph, child-job of some jobflow or worker for distributed transformation in cluster. When the execution doesn't have a parent, the

ROOT_RUN_ID is the same as RUN_ID.

Username of user who launched the graph or jobflow.

Id of node running the graph or jobflow.

140

Chapter 19. Graph/

Jobflow Parameters

Parameters by Execution Type

Additional parameters are passed to the graph depending on how the graph is executed.

Executed from Web GUI

Graphs executed from a web gui have no additional parameters.

Executed by Launch Service Invocation

Service parameters which have Pass to graph attribute enabled are passed to the graph not only as "dictionary" input data, but also as graph parameter.

Executed by HTTP API Run Graph Operation Invocation

Any URL parameter with "param_" prefix is passed to executed graph but without "param_" prefix. i.e.

"param_FILE_NAME" specified in URL is passed to the graph as property named "FILE_NAME".

Executed by RunGraph Component

Since 3.0 only parameters specified by "paramsToPass" attribute are passed from the "parent" graph to the executed graph. However common properties (RUN_ID, PROJECT_DIR, etc.) are overwritten with new values.

Executed by WS API Method executeGraph Invocation

Parameters with values may be passed to the graph with the request for execution.

Executed by Task "graph execution" by Scheduler

Table 19.2. passed parameters

key

EVENT_SCHEDULE_EVENT_TYPE

EVENT_SCHEDULE_LAST_EVENT

EVENT_SCHEDULE_DESCRIPTION

EVENT_USERNAME

EVENT_SCHEDULE_ID

description

Type of a schedule: SCHEDULE_PERIODIC |

SCHEDULE_ONETIME

Date/time of a previous event

Schedule description, which is displayed in web GUI

User who "owns" the event. For schedule it is the user who created the schedule.

ID of schedule which triggered the graph

Executed from JMS Listener

There are many graph parameters and dictionary entries passed, depending on the type of incoming message. See

details in JMS Messages Listeners (p. 187).

Executed by Task "Start a graph" by Graph/Jobflow Event Listener

Since 3.0 only specified properties from a "source" job are passed to executed job by default. There is a

"graph.pass_event_params_to_graph_in_old_style" server config property which can change this behavior so that

141

Chapter 19. Graph/

Jobflow Parameters

ALL parameters from a "source" job are passed to the executed job. This switch is implemented for backwards compatibility. Regarding to the default behaviour: in the editor of graph event listener, you can specify a list of

parameters to pass. Please see Start a Graph (p. 152) for details.

The following parameters with current values are always passed to the target job

Table 19.3. passed parameters

key

EVENT_RUN_SANDBOX

EVENT_JOB_EVENT_TYPE

EVENT_RUN_JOB_FILE

EVENT_RUN_ID

EVENT_TIMEOUT

EVENT_RUN_RESULT

EVENT_USERNAME

description

Sandbox with the graph which is source of the event

GRAPH_STARTED | GRAPH_FINISHED |

GRAPH_ERROR | GRAPH_ABORTED |

GRAPH_TIMEOUT

GRAPH_STATUS_UNKNOWN,

JOBFLOW_* for jobflow event listeners.

analogically

| jobFile of the job which is source of the event

ID of the graph execution which is source of the event.

Number of milliseconds which specifies interval of timeout. Makes sense only for "timeout" graph event.

Result (or current status) of the execution which is source of the event.

User who "owns" the event. For graph events it is the user who created the graph event listener

Executed by Task "graph execution" by File Event Listener

Table 19.4. passed parameters

key

EVENT_FILE_PATH

EVENT_FILE_NAME

EVENT_FILE_URLS

description

Path to the file which is source of the event. Does not contain a file name. Does not end with a file separator.

Is passed only for the local file event listener.

Filename of the file which is source of the event. Is passed only when the "grouping" mode is disabled.

Otherwise there are more file events, not just one.

Contans string, which may be used "as is" in the

"file URL" attribute of various CloverETL components.

It may contain URL to one or more (if grouping is anabled) files. It may contain local path(s) or remote

URL(s) where credentials are replaced by placeholders

(due to security reasons).

EVENT_FILE_AUTH_USERNAME Username/ID to the remote location.

EVENT_FILE_AUTH_USERNAME_URL_ENCODEDThe same as EVENT_FILE_AUTH_USERNAME, but the value is also URL encoded, so it may be used in the

URL.

EVENT_FILE_AUTH_PASSWORD Password/key to the remote location. It's encrypted by the master password. It's passed only when the file listener uses user+password authentication.

EVENT_FILE_AUTH_PASSWORD_URL_ENCODEDThe same as EVENT_FILE_AUTH_PASSWORD, but the value is also URL encoded, so it may be used in the

URL (EVENT_FILE_URLS parameter).

142

key

EVENT_FILE_EVENT_TYPE

EVENT_FILE_PATTERN

EVENT_FILE_LISTENER_ID

EVENT_USERNAME

Chapter 19. Graph/

Jobflow Parameters

description

SIZE | CHANGE_TIME | APPEARANCE |

DISAPPEARANCE

Pattern specified in a file event listener

User who "owns" the event. For file events, it is the user who created the file event listener.

Adding Another Graph Parameters

Additional "Graph Config Parameters"

It is possible to add so-called additional parameters in Web GUI - section Sandboxes for the selected graph or for

all graphs in the selected sandbox. See details in Job Config Properties (p. 119).

Task "execute_graph" Parameters

The "execute graph" task may be triggered by schedule, graph event listener, or file event listener. Task editor allows you to specify key=value pairs which are passed to executed graph.

143

Chapter 20. Tasks

Task is a graph, jobflow, groovy script, etc. that can be started manually, started on scheduled time, or triggered by some event. Task basically specifies WHAT to do.

There are several tasks implemented for schedule and for graph event listener as follows:

•

•

•

•

•

•

•

•

•

Chapter 22, Scheduling (p. 165)

Tasks in Cluster Environment

In the Cluster environment, you can specify node where the task runs. The task can run on Any node or on one

of selected nodes. If there is no node ID specified, task may be processed on any cluster node, so in most cases it will be processed on the same node where the event was triggered. If there are some nodeIDs specified, task will be processed on the first node in the list which is connected in cluster and ready.

Tasks are used in

Chapter 24, Listeners (p. 176)

Chapter 21, Manual Task Execution (p. 164)

144

Chapter 20. Tasks

Send an Email

The "send e-mail" task is useful for notifications about result of graph execution. E.g., you can create a listener with this task type to be notified about each failure in specified sandbox or failure of particular graph.

This task is very useful, but for now only as response for graph events. This feature is very powerful for monitoring.

(see

Graph Event Listeners (p. 177) for description of this task type).

Note: It seems useless to send e-mails periodically, but it may send current server status or daily summary. These features will be implemented in further versions.

Table 20.1. Attributes of "Send e-mail" task

Task type

Bcc

Reply-to (Sender)

Subject

HTML

Text

Log file as attachment

"Send an email"

Recipient's e-mail address. It is possible to specify more addresses separated by a comma. It is also possible to use placeholders. See

is also possible to use placeholders. See Placeholders

Cc stands for 'carbon copy'. Copy of the e-mail will be delivered to these addresses. It is possible to specify more addresses separated by a comma. It

Bcc: stands for 'Blind carbon copy'. It is the same as Cc, but the others recipients aren't aware, that these recipients get copy of the e-mail.

E-mail address of sender. It must be a valid address according to SMTP server. It is also possible to use placeholders. See

(p. 146)

for details.

E-mail subject. It is also possible to use placeholders. See

A body of the e-mail in HTML. The e-mail is created as multipart, so

HTML body should have a precedence. A plain text body is only for e-mail clients which do not display HTML. It is also possible to use placeholders.

See

A body of the e-mail in plain text. The e-mail is created as multipart, so

HTML body should have a precedence. A plain text body is only for e-mail clients which do not display HTML. It is also possible to use placeholders.

See

Note: Do not forget to configure connection to SMTP server (See Part III, “

If this switch is checked, e-mail will have an attachment with a packed log file of the related graph execution.

145

Chapter 20. Tasks

Figure 20.1. Web GUI - send e-mail

Configuration

” (p. 42) for details).

Placeholders

Placeholder may be used in some fields of tasks. They are especially useful for e-mail tasks, where you can generate content of e-mail according to context variables.

Note: In most cases, you can avoid this by using e-mail templates (See E-mail task for details)

These fields are preprocessed by Apache Velocity templating engine. See Velocity project URL for syntax description http://velocity.apache.org/

There are several context variables, which you can use in placeholders and even for creating loops and conditions.

• event

• now

• user

• run

• sandbox

146

Chapter 20. Tasks

Some of them may be empty depending on type of event. E.g., if task is processed because of graph event, then

run and sandbox variables contain related data, otherwise they are empty.

Table 20.2. Placeholders useful in e-mail templates

event now task user

Variable name

run sandbox tracking

Contains

Event that has triggered the task.

Current date-time

The triggered task.

User, who caused this event. It may be an owner of a schedule, or someone who executed a graph. It contains sub-properties, which are accessible using dot notation

(i.e. ${user.email}) e-mail:

• user.email

• user.username

• user.firstName

• user.lastName

• user.groups (list of values)

Data structure describing one single graph execution. It contains sub-properties, which are accessible using dot notation (i.e. ${run.jobFile})

• run.jobFile

• run.status

• run.startTime

• run.stopTime

• run.errNode

• run.errMessage

• run.errException

• run.logLocation

Data structure describing a sandbox containing the executed graph. It contains subproperties, which are accessible using dot notation (i.e. ${sandbox.name})

• sandbox.name

• sandbox.code

• sandbox.rootPath

A data structure describing status of components in graph execution. It contains subproperties, which are accessible using Velocity syntax for loops and conditions.

#if (${tracking})

#foreach ($phase in $tracking.trackingPhases)

<tr><td>phase: ${phase.phaseNum}</td>

<td>${phase.executionTime} ms</td>

#foreach ($node in $phase.trackingNodes)

<tr><td>${node.nodeName}</td>

<td>${node.result}</td>

#foreach ($port in $node.trackingPorts)

<td>${port.type}:${port.index}</td>

<td>${port.totalBytes} B</td>

<td>${port.totalRows} rows</td></tr>

#end

</table>

#end

}

147

Variable name

schedule parameters

Chapter 20. Tasks

Contains

Data structure describing schedule which triggered this task. It contains subproperties, which are accessible using dot notation (i.e. ${schedule.description})

Available in scheduling.

• schedule.description

• schedule.startTime

• schedule.endTime

• schedule.lastEvent

• schedule.nextEvent

• schedule.fireMisfired

Task parameters - container for String-String key-value pairs passed to this task. In manual task execution.

148

Chapter 20. Tasks

Execute Shell Command

Execute Shell Command executes system command or a shell script.

This task is used is used in Chapter 22, Scheduling

(p. 165), Chapter 24, Listeners

(p. 176), and Chapter

21, Manual Task Execution (p. 164).

Table 20.3. Attributes of "Execute shell command" task

Task type

Start on

Shell script

Working directory

Timeout

"Execute shell command"

Node IDs to process the task

This attribute is accessible only in the cluster environment. If there are nodes specified, the task will be processed on the first node which is online and ready.

Command line for execution of external process.

Working directory for process.

If not set, working directory of application server process is used.

Timeout in milliseconds. After period of time specified by this number, external process is terminated and all results are logged.

Figure 20.2. Web GUI - shell command

149

Chapter 20. Tasks

Execute Shell Command Parameters

Some parameters are available only in particular context: scheduling, event listeners, or manual task execution.

Table 20.4. Parameters of "Execute shell command" task

event now task user

Event that has triggered the task

Current date-time

The triggered task

Object representing user who executed the graph/jobflow. It contains subproperties that are accessible using dot notation (i.e. ${user.email})

• user.email

• user.username

• user.firstName

• user.lastName

• user.groups (list of values)

Table 20.5. Parameters of "Execute shell command" task - available in scheduling

schedule

EVENT_USERNAME

EVENT_USER_ID

EVENT_SCHEDULE_DESCRIPTION

EVENT_SCHEDULE_EVENT_TYPE

EVENT_SCHEDULE_ID

EVENT_SCHEDULE_LAST_EVENT

Object representing the schedule that triggered this task. It contains sub-properties that are accessible using dot notation (i.e.

${schedule.description})

• schedule.description

• schedule.startTime

• schedule.endTime

• schedule.lastEvent

• schedule.nextEvent

• schedule.fireMisfired

Username of the user who caused the event.

Numeric ID of the user who caused the event.

Description of the schedule.

Type of the schedule - SCHEDULE_ONETIME or

SCHEDULE_PERIODIC.

Numeric ID of the schedule

Date-time of the latest schedule triggering (in java.util.Date.toString() format).

Table 20.6. Parameters of "Execute shell command" task - available in listeners

run Object representing single graph/jobflow execution. It contains sub-properties that are accessible using dot notation (i.e. ${run.jobFile}).

• run.jobFile

• run.status

• run.startTime

• run.stopTime

• run.errNode

• run.errMessage

• run.errException

150

Chapter 20. Tasks sandbox Object representing a sandbox containing the executed graph/jobflow.

It contains sub-properties that are accessible using dot notation (i.e.

${sandbox.name}) tracking

EVENT_USERNAME

EVENT_USER_ID

EVENT_RUN_SANDBOX

EVENT_RUN_JOB_FILE

EVENT_RUN_RESULT

• sandbox.name

• sandbox.code

• sandbox.rootPath

Object represeting status of components in a graph execution. It contains subproperties that are accessible using Velocity syntax for loops and conditions.

Username of the user who caused the event.

Numeric ID of the user who caused the event.

Code of the sandbox containing the graph/jobflow.

Sandbox-relative path to the graph/jobflow file.

Current status of the graph/jobflow execution

• N_A

• READY

• RUNNING

• WAITING

• FINISHED_OK

• ERROR

• ABORTED

• TIMEOUT

• UNKNOWN

EVENT_RUN_ID

EVENT_TIMEOUT

Numeric ID of the run record representing graph/jobflow execution

Specified timeout (in milliseconds) for the TIMEOUT event to occur.

EVENT_JOB_EVENT_TYPE Graph event that triggered the task

• GRAPH_STARTED

• GRAPH_PHASE_FINISHED

• GRAPH_FINISHED

• GRAPH_ERROR

• GRAPH_ABORTED

• GRAPH_TIMEOUT

• GRAPH_STATUS_UNKNOWN

Table 20.7. Parameters of "Execute shell command" task - available in manual task execution

parameters Task parameters - container for String-String key-value pairs passed to this task.

151

Chapter 20. Tasks

Start a Graph

Start a Graph starts a specified graph from specified sandbox.

Table 20.8. Attributes of "Graph execution" task

Task type

Start on

Sandbox

Graph

Save run record

Parameters

"Start a graph"

Node(s) to process the task. It is a cluster feature.

This select box contains sandboxes which are readable for logger user.

Select sandbox which contains graph to execute.

The graph to be executed.

This select box is filled with all graphs files accessible in selected sandbox.

Type a graph name or path to filter available items.

Saves run record to database.

If the task runs too often (once in several seconds), you can increase the database performance by disabling this.

List of parameters passed to the graph.

Event parameters like "EVENT_RUN_RESULT", "EVENT_RUN_ID" etc. are passed to the executed job without limitations. Parameters

EVENT_RUN_RESULT and EVENT_RUN_ID are used in context of event listeners. They are not used in context of scheduling.

Figure 20.3. Web GUI - Graph execution task

Please note that behaviour of this task type is almost the same as Start a Jobflow (p. 154).

152

Chapter 20. Tasks

Parameters

You can start a graph with parameters.

To start a graph with parameter choose an existing parameter from a list, set its value, and click the plus sign button at the end of line.

Figure 20.4. Web GUI - Graph execution task

If the graph is started by event listener, it receives some additional parameters from the triggering job.

Parameters passed to graph by Event Listeners

Table 20.9. Additional parameters available in Event Listeners

EVENT_USERNAME

EVENT_USER_ID

EVENT_RUN_SANDBOX

EVENT_RUN_JOB_FILE

EVENT_RUN_RESULT

Username of the user who caused the event

Numeric ID of the user who caused the event.

Code of the sandbox containing the graph/jobflow

Sandbox-relative path to the graph/jobflow file.

Current status of the graph/jobflow execution (N_A, READY,

RUNNING, WAITING, FINISHED_OK, ERROR, ABORTED, TIMEOUT or UNKNOWN).

EVENT_RUN_ID Numeric ID of the run record representing graph/jobflow execution.

EVENT_JOB_EVENT_TYPE Graph/Jobflow event type that triggered the task.

EVENT_TIMEOUT Specified timeout (in milliseconds) for the TIMEOUT event to occur

153

Chapter 20. Tasks

Start a Jobflow

Start a jobflow starts a specified jobflow from a specified sandbox.

Table 20.10. Attributes of "Jobflow execution" task

Task type

Start on

Sandbox

Jobflow

Save run record

Parameters

"Start a jobflow"

Node(s) to process the task

This attribute is accessible only in the cluster environment. If there are nodes specified, the task will be processed on the first node which is online and ready.

This select box contains sandboxes which are readable for logger user.

Select sandbox which contains jobflow to execute.

This select box is filled with all jobflow files accessible in selected sandbox.

Type jobflow name or path to filter available items.

Saves run record to database. If the task runs too often (once in several seconds), you can increase the database performance by disabling this.

Key-value pairs which are passed to the executed job as parameters.

Event parameters like "EVENT_RUN_RESULT", "EVENT_RUN_ID" etc. are passed to the executed job without limitations. Parameters

EVENT_RUN_RESULT and EVENT_RUN_ID are used in context of event listeners. They are not used in context of scheduling.

Figure 20.5. Web GUI - Jobflow execution task

Please note that behaviour of this task type is almost the same as Start a Graph (p. 152).

If the jobflow start is triggered by an event, the same set of parameters as in graph event listener is passed to

jobflow. Parameters (p. 153).

154

Chapter 20. Tasks

Start a Profiler Job

Start a profiler job starts a specified profiler job from a specified directory.

You can pass parameters to the profiler job in the same way as in case you

Start a Graph (p. 152) or

Start a

Jobflow (p. 154).

In case of triggering the profiler job by event listener, the same set of additional parameters, as in case of execution of graph, is passed to the profiler job. See

Parameters passed to graph by Event Listeners (p. 153).

155

Chapter 20. Tasks

Abort job

This task kills/aborts specified job (ETL graph or jobflow), if it is currently running.

Table 20.11. Attributes of "Abort job" task

Task type

Start on

Kill source of event

Sandbox

Job

"Abort job"

Node(s) to process the task.

This attribute is accessible only in the cluster environment. If there are nodes specified, the task will be processed on the first node which is online and ready.

If this switch is on, task will kill the job which is source of the event, which activated this task. Attributes sandbox and job are ignored.

This checkbox makes sense only if Abort job is activated by some event.

Select a sandbox which contains the job to kill.

This attribute works only when "Kill source of event" switch is off.

This select box is filled with all jobs accessible in selected sandbox. All instances of selected job that are currently running and will be killed. This attribute works only when "Kill source of event" switch is off.

Figure 20.6. Web GUI - "Abort job"

156

Chapter 20. Tasks

Archive Records

This task can archive (or delete) obsolete records from DB.

Table 20.12. Attributes of "Archivator" task

Task type

Start on

Archivator type

Older than

Output path for archives

Include executions history

Include temp files

Include tasks history

Include profiler runs

Include server instance history

"Archivator"

This attribute specifies cluster node on which may process the task.

This attribute is accessible only in the cluster environment.

If it's empty, it may be any node, if there are nodes specified, the task will be processed on the first node which is online and ready.

There are two possible values: "archive" or "delete".

Delete option removes records without any possibility of UNDO operation.

Archive option removes records from DB, but creates a ZIP package with

CSV files containing deleted data.

Time period (in minutes) - it specifies which records are evaluated as obsolete. Records older than the specified interval are stored in archives.

This attribute makes sense only for "archive" type.

Includes executions history into archived file.

If checked, archivator removes all graph temporary files older than the given timestamp defined in "Older than" attribute. The temporary files are files with graph debug data, dictionary files and files created by graph components.

If checked, archivator will include run records. Log files of graph runs are included as well.

If checked, archivator will include profiler job results.

Includes server instance history into archived file.

157

Chapter 20. Tasks

Figure 20.7. Web GUI - archive records

158

Chapter 20. Tasks

Send a JMS Message

This type of task is useful for notifications about result of graph execution. E.g. you can create a graph event listener with this task type to be notified about each failure in a specific sandbox or failure of a particular graph.

JMS messaging requires JMS API (jms.jar) and third-party libraries. All these libraries must be available on application server classpath. Some application servers contain these libraries by default, some do not, thus the libraries must be added explicitly.

Table 20.13. Attributes of JMS message task

Task type

Initial context

Initial context factory class

Broker URL

Connection factory

Queue/Topic

Username

Password

JMS pattern

Text

"JMS message"

Choose between default and custom initial context.

A full class name of javax.naming.InitialContext implementation. Each

JMS provider has its own implementation. E.g., for Apache MQ it is

"org.apache.activemq.jndi.ActiveMQInitialContextFactory". If it is empty, server uses the default initial context.

URL of a JMS message broker

A JNDI name of a connection factory. It depends on a JMS provider.

JNDI name of a message queue/topic on the server

Username for connection to a JMS message broker

Password for connection to a JMS message broker

This select box is available only when user is creating a new record. It contains all predefined JMS message patterns. If user chooses any of them, text field below is automatically filled with value from pattern.

Body of a JMS message. It is also possible to use placeholders. See

(p. 146) of send e-mail task for details.

159

Chapter 20. Tasks

Figure 20.8. Web GUI - Task JMS message editor

Table 20.14. Parameters of "Send a JMS Message"

event now task user schedule

EVENT_USERNAME

EVENT_USER_ID

EVENT_SCHEDULE_DESCRIPTION

Event that has triggered the task.

Current date-time

The triggered task.

Object representing owner of the schedule. It contains subproperties that are accessible using dot notation (i.e. ${user.email}) e-mail, username, firstName, lastName, groups (list of values).

Object representing the schedule that triggered this task. It contains sub-properties that are accessible using dot notation

(i.e. ${schedule.description}) description, startTime, endTime, lastEvent, nextEvent, fireMisfired.

Username of the user who caused the event

Numeric ID of the user who caused the event.

Description of the schedule

160

Chapter 20. Tasks

EVENT_SCHEDULE_EVENT_TYPE

EVENT_SCHEDULE_ID

EVENT_SCHEDULE_LAST_EVENT

Type of the schedule - SCHEDULE_ONETIME or

SCHEDULE_PERIODIC.

Numeric ID of the schedule.

Date-time of the latest schedule triggering (in java.util.Date.toString() format).

161

Chapter 20. Tasks

Execute Groovy Code

This type of task allows you to execute code written in scripting language Groovy. The script can be defined in place or using a path to external

.groovy

file. It is possible to use some variables.

Basic attribute of this task is source code written in Groovy.

If source codes are provided from both a file and through the input form only the code from the input form will be executed.

In cluster environment there is also one additional attribute "Node IDs to process the task". If it's empty, it may be any node, if there are nodes specified, the task will be processed on the first node which is online and ready.

CloverETL Server contains Groovy version 2.0.0

Table 20.15. List of variables available in Groovy code

variable

event

class description availability

every time task now parameters user com.cloveretl.server.

persistent.Task

java.util.Date

java.util.Properties

com.cloveretl.server.

persistent.User

com.cloveretl.server.

persistent.RunRecord

task.

The triggered task.

current time

Properties of a task

Same as event.getUser() every time every time every time every time run sandbox tracking schedule servletContext com.cloveretl.server.

persistent.Sandbox

com.cloveretl.server.

persistent.TrackingGraph

com.cloveretl.server.

persistent.Schedule

same as run.getTrackingGraph() same as

((ScheduleServerEvent)event).

getSchedule()

When the event is instance of

GraphServerEvent

When the event is instance of

ScheduleServerEvent javax.servlet.ServletContext Servlet context cloverConfiguration com.cloveretl.server.spring.

CloverConfiguration serverFacade com.cloveretl.server.facade.

api.ServerFacade

The run record.

same as run.getSandbox()

Configuration values for

CloverETL Server

Reference to the facade interface. Useful for calling

CloverETL Server core.

When the event is instance of

GraphServerEvent

When the event is instance of

GraphServerEvent every time every time every time sessionToken String

WAR file contains JavaDoc of facade API and it is accessible on URL: http://host:port/ clover/javadoc/index.html

Valid session token of the user who owns the event. It is every time

162

Chapter 20. Tasks

variable class description

useful for authorisation to the facade interface.

availability

Variables run, tracking and sandbox are available only if event is instance of GraphServerEvent class. Variable schedule is only available for ScheduleServerEvent as event variable class.

Example of use Groovy script

This example shows a script which writes text file describing the finished graph. It shows use of 'run' variable.

import com.cloveretl.server.persistent.RunRecord;

String dir = "/tmp/";

RunRecord rr = (RunRecord)run;

String fileName = "report"+rr.getId()+"_finished.txt";

FileWriter fw = new FileWriter(new File(dir+fileName)); fw.write("Run ID :"+rr.getId()+"\n"); fw.write("Graph ID :"+rr.getGraphId()+"\n"); fw.write("Sandbox :"+rr.getSandbox().getName()+"\n"); fw.write("\n"); fw.write("Start time :"+rr.getStartTime()+"\n"); fw.write("Stop time :"+rr.getStopTime()+"\n"); fw.write("Duration :"+rr.getDurationString()+"\n"); fw.write("Status :"+rr.getStatus()+"\n"); fw.close();

163

Chapter 21. Manual Task Execution

Since 3.1

Manual task execution allows you to invoke a task directly with an immediate effect, without defining and triggering an event.

There are a number of task types that are usually associated with a triggering event, such as a file listener or a graph/jobflow listener. You can execute any of these tasks manually.

Additionally, you can specify task parameters to simulate a source event that would normally trigger the task. The following is a figure displaying how a “file event” could be simulated. The parameters for various event sources are listed in the section "Graph parameters".

Figure 21.1. Web GUI - "Manual task execution" form

Using Manual Task Execution

In server GUI, switch to Event Listeners tab. In the New Listener drop-down menu, select the Manual Task

Execution option.

Choose the task type you would like to use. See also documenation on chosen tasks:

Manual task execution permission (p. 105).

To access the Manual Task Execution form, you need

164

Chapter 22. Scheduling

The scheduling module allows you to create a time schedule for operations you need to trigger in a repetitive or timely manner.

Similar to “cron” from Unix systems, each schedule represents a separate time schedule definition and a task to perform.

In the Cluster, you can explicitly specify which node should execute the scheduled task using the Node ID parameter. However, if not set, the node will be selected automatically from all available nodes (but always just one).

Figure 22.1. Web GUI - section "Scheduling" - create new

The tasks you can schedule are described in Chapter 20, Tasks (p. 144).

Onetime Schedule (p. 166)

165

Chapter 22. Scheduling

Timetable Setting

This section should describe how to specify when schedule should be triggered. Please keep in mind, that exact trigger times are not guaranteed. There may be couple of seconds delay. Schedule itself can be specified in different ways.

•

•

Periodical schedule by Interval (p. 168)

•

Periodical schedule by timetable (Cron Expression) (p. 169)

Onetime Schedule

This schedule is triggered just once.

Table 22.1. Onetime schedule attributes

Type

Start date/time

"onetime"

Date and time, specified with minutes precision.

Fire misfired event as soon as possible

If checked and trigger time is missed because of any reason (i.e. server restart), it will be triggered immediately, when it is possible. Otherwise it is ignored and it will be triggered at next scheduled time.

Figure 22.2. Web GUI - onetime schedule form

166

Chapter 22. Scheduling

Figure 22.3. Web GUI - schedule form - calendar

167

Chapter 22. Scheduling

Periodical schedule by Interval

This type of schedule is the simplest periodical type. Trigger times are specified by these attributes:

Table 22.2. Periodical schedule attributes

Type

Periodicity

Not active before date/time

"periodic"

"interval"

Date and time, specified with minutes precision.

Not active after date/time

Interval (minutes)

Date and time, specified with minutes precision.

Specifies interval between two trigger times. Next task is triggered even if previous task is still running.

Fire misfired event as soon as possible

Figure 22.4. Web GUI - periodical schedule form

168

Chapter 22. Scheduling

Periodical schedule by timetable (Cron Expression)

Timetable is specified by powerful (but a little bit tricky) cron expression.

Table 22.3. Cron periodical schedule attributes

Type

Periodicity

Not active before date/time

Not active after date/time

"periodic"

"by timetable"

Date and time, specified with minutes precision.

Cron expression Cron is powerful tool, which uses its own format for scheduling. This format is well known among UNIX administrators. i.e. "0 0/2 4-23 * * ?" means "every 2 minutes between 4:00am and 11:59pm".

Fire misfired event as soon as possible

If checked and trigger time is missed because of any reason (i.e. server restart), it will be triggered immediately when it is possible. Otherwise it is ignored and it will be triggered at next scheduled time.

Figure 22.5. Cron periodical schedule form

The Edit button helps you to set up cron expression even without knowledge of the exact cron expression format.

Figure 22.6. Editing the cron expression - minutes

169

Chapter 22. Scheduling

Figure 22.7. Editing the cron expression - days of week

170

Chapter 22. Scheduling

Allocations of Scheduled Task on Nodes

In cluster environment, you can set node on which the scheduled task will be launched.

The task can be scheduled on arbitrary node or on one or more specified nodes.

Allocation on Any Arbitrary Node

Figure 22.8. Schedule allocation - Any node

Allocation on One or More Chosen Nodes

Figure 22.9. Schedule allocation - One ore more specific nodes

171

Chapter 22. Scheduling

Scheduling the Tasks - Examples

Start a graph at specific time

This example shows scheduling the start of a graph at the specific time.

1. In server GUI, in Scheluling section, click New schedule.

2. Enter the description that allows you to successfully identify the task between other existing tasks, e.g. My

one-time processing example.

3. Choose onetime to start the graph just once.

4. Enter start date and time. The calendar accessible under calendar icon might help you to enter the required date and time in correct format.

5. Choose Task Start a graph.

6. Select the sandbox and graph within the sandbox.

Start a Jobflow once an hour

This example show scheduling the repetitive run.

Create a new schedule that runs UserStats.jbf jobflow from reports sandbox once an hour.

1. Enter description, e.g. Hourly user stats.

2. Choose periodic type and periodicity by interval.

3. The task will be started once an hour within some time period. Enter the beginning of this period (not active

before) and end of the period (not active after).

4. Enter interval between two jobflow starts.

5. Select the sandbox and graph within the sandbox.

Complex Scheduling

This example shows complex scheduling using cron expression.

Start a graph WeekendGraph.grf every Saturday and Sunday at 22:15.

1. Enter the description of the schedule, e.g. The Weekend Graph

2. Select periodic type and periodicity by timetable.

3. Click Edit on line starting with Cron expression.

On Minutes tab, select Every selected minute and select 15.

On Hours tab, select Every selected hour and select 22.

On Days of Week tab, select Every selected day of week and click Saturday and Sunday.

4. The Task is Start a graph

5. Select the sandbox and graph.

172

Chapter 23. Viewing Job Runs - Executions History

Executions History shows the history of all jobs that the Server has executed – transformation graphs, jobflows, and Data Profiler jobs. You can use it to find out why a job failed, see the parameters that were used for a specific run, and much more.

The table shows basic information about the job: Run ID, Job file, current status, and time of execution, as well as some useful links. You will also find additional details after clicking on the job name in the list – details such as associated log files, parameter values, tracking, and more.

Please note that some jobs might not appear in the Executions History list. These are jobs that have disabled persistency for increased performance – e.g. some Launch Services disable storing the run information in order to increase service responsiveness.

Filtering and ordering

Use the Filter panel to filter the view. By default, only parent tasks are shown (Show executions children) – e.g.

master nodes in the Cluster and their workers are hidden by default.

Use the up and down arrows in the table header to sort the list. By default, the latest job is listed first.

Figure 23.1. Executions History - executions table

When some job execution is selected in the table, the detail info is shown on the right side.

173

Chapter 23. Viewing Job

Runs - Executions History

Table 23.1. Persistent run record attributes

Run ID

Attribute

Execution type

Parent run ID

Root run ID

Execution group

Description

A unique number identifying the run of the job. Server APIs usually return this number as a simple response to the execution request. It's useful as a parameter of subsequent calls for specification of the job execution.

Type of a job as recognized by the server. STANDALONE for ETL graph,

JOBFLOW for Jobflow, PROFILER_JOB for profiler, MASTER for main record of partitioned execution in cluster, PARTITIONED_WORKER for worker record of partitioned execution in cluster

Run ID of the parent job. Typically Jobflow which executed this job, or master execution which encapsulates this worker execution.

Run ID of the root parent job. Job execution which wasn't executed by another parent job.

Jobflow components may group sub-jobs using this attribute. See a description of

Jobflow components for details.

Indication that this job execution has or has not any child execution.

In cluster mode shows ID of the cluster node which this execution was running on.

Nested jobs

Node

Executed by

Sandbox

Job file

Job version

User which executed the job. Either directly using some API/GUI or indirectly using the scheduling or event listeners.

Sandbox containing a job file. For jobs which are sent together with an execution request, so the job file doesn't exist on the server site, it's set to "default" sandbox.

Path to a job file, relative to the sandbox root. For jobs which are sent together with an execution request, so the job file doesn't exist on the server site, it's set to generated string.

Revision of the job file. It's a string generated by CloverETL Designer and stored in the job file.

Status

Error message

Exception

Input parameters

Status of the job execution. READY - waiting for execution start, RUNNING processing job, FINISHED OK - job finished without any error, ABORTED - job was aborted directly using some API/GUI or by parent Jobflow, ERROR - job failed,

N/A (not available) - server process died suddenly, so it couldn't properly abort the jobs, so after restart the jobs with unknown status are set as N/A

Server date-time (and timezone) of the execution start.

Started

Finished

Duration

Server date-time (and timezone) of the execution finish.

Execution duration

Error in component ID If the job failed due the error in a component, this field contains ID of the component.

Error in component type If the job failed due the error in a component, this field contains type of the component.

If the job failed, this field contains error description.

If the job failed, this field contains error stack trace.

Input dictionary

List of input parameters passed to the job. Job file can't be cached, since the parameters are applied during loading from the job file. Job file isn't cached by default.

List of dictionary elements passed to the job. Dictionary is used independently of job file caching.

Output dictionary List of dictionary elements at the moment the job ends.

For jobs which have some children executions, e.g. partitioned or jobflows also an executions hierarchy tree is shown.

174

Chapter 23. Viewing Job

Runs - Executions History

Figure 23.2. Executions History - overall perspective

Since the detail panel and expecially job logs may be wide, it may be useful to hide a table on the left, so the detail panel spreads. Click on the minimize icon on the top of the list panel to hide the panel. Then to show list panel again, click to the "Expand panel" icon on the left.

Figure 23.3. Executions Hierarchy with docked list of jobs

Executions hierarchy may be rather complex, so it's possible to filter the content of the tree by a fulltext filter.

However when the filter is used, the selected executions aren't hierarchically structured.

175

Chapter 24. Listeners

Listeners can be seen as hooks. They wait for a specific event and take a user-defined action if the event occurs.

The event is specific to the particular listener

Figure 24.1. Listeners

Graph Event Listeners (p. 177)

Jobflow Event Listeners (p. 184)

JMS Messages Listeners (p. 187)

Universal Event Listeners (p. 192)

File Event Listeners (remote and local) (p. 194)

Task Failure Listeners (p. 200)

The available actions taken by the listeners are common for all listeners. The actions that can be taken are:

• Send an e-mail - see Send an Email (p. 145)

• Execute a shell command - see Execute Shell Command (p. 149)

• Start a graph - see Start a Graph (p. 152)

• Start a jobflow - see

• Start a profiler job

• Abort a job - see

• Archivator - see Archive Records (p. 157)

• Send a JMS Message - see Send a JMS Message (p. 159)

• Execute a Groovy code - see

Jobflow Event Listeners (p. 184)) – for

176

Chapter 24. Listeners

Graph Event Listeners

Graph Event Listeners allow you to define a task that the Server will execute as a reaction to the success, failure or other event of a specific job (a transformation graph).

Each listener is bound to a specific graph and is evaluated no matter whether the graph was executed manually, scheduled, or via an API call, etc.

You can use listeners to chain multiple jobs (creating a success listener that starts the next job in a row). However, we recommend using Jobflows to automate complex processes because of its better development and monitoring capabilities.

Graph Event Listeners are similar to Jobflow Event Listeners (

CloverETL Server both are simply “jobs”.

In the Cluster, the event and the associated task are executed on the same node the job was executed on by default.

If the graph is distributed, the task will be executed on the master worker node. However, you can override where the task will be executed by explicitly specifying a Node IDs in the task definition.

Graph Events

Each event carries properties of a graph, which is source of the event. If there is an event listener specified, task may use these properties. E.g. next graphs in the chain may use "EVENT_FILE_NAME" placeholder which was activated by the first graph in the chain. Graph properties, which are set specifically for each graph run (i.e.

RUN_ID), are overridden by the last graph.

Types of graph events

•

Graph started (p. 177)

•

Graph phase finished (p. 177)

•

Graph finished (p. 177)

•

Graph error (p. 177)

•

Graph aborted (p. 178)

•

Graph timeout (p. 178)

•

Graph unknown status (p. 178)

Graph started

Graph started event is created, when ETL graph execution successfully started.

Graph phase finished

Graph phase finished event is created, everytime when graph phase is finished and all its nodes are finished with status

FINISHED_OK

Graph finished

Graph finished event is created, when all phases and nodes of graph are finished with

FINISHED_OK

status.

Graph error

Graph error event is created, when graph cannot be executed for any reason, or when any node of graph fails.

177

Chapter 24. Listeners

Graph aborted

Graph aborted event is created, when graph is explicitly aborted.

Graph timeout

Graph timeout event is created, when graph runs longer than for a specified interval. Thus you should specify a

"Job timeout interval" attribute for each listener of a graph timeout event. You can specify this interval in seconds or in minutes or in hours.

Figure 24.2. Web GUI - graph timeout event

Graph unknown status

Graph unknown status event is created, when the server, during the startup, detects run records with undefined status in the executions history. Undefined status means, that server has been killed during the graph run. Server automatically changes state of a graph to Not Available and sends a graph unknown status event.

Please note, that this works just for executions, which have a persistent record in executions history. It is possible to execute a transformation without persistent record in executions history, typically for better performance of fast running transformations (i.e. using Launch Services).

Listener

User may create a listener for a specific event type and graph (or all graphs in sandbox). The listener is actually connection between graph event and the task, where graph event specifies when and task specifies what to do.

Event handling consist of the following course of actions:

• event is created

178

Chapter 24. Listeners

• listeners for this event are notified

• each listener performs related task

Tasks

Task types are described in Chapter 20, Tasks (p. 144), see this chapter for details about these task types.

In the Cluster environment, all tasks have an additional attribute "Node IDs to process the task". If there is no node

ID specified, the task may be processed on any cluster node. In most cases it will be processed on the same node where the event was triggered. If there are some nodeIDs specified, the task will be processed on the first node in the list which is connected in cluster and ready.

Execute graphs in chain (p. 179)

Use Cases

Possible use cases are the following:

•

•

Email notification about graph failure (p. 181)

•

Email notification about graph success (p. 182)

•

Backup of data processed by graph (p. 183)

Execute graphs in chain

For example, we have to execute graph B, only if another graph A finished without any error. So there is some kind of relation between these graphs. We can achieve this behaviour by creating a graph event listener. We create a listener for graph finished

event of graph A and choose an start a graph

task type with graph

B specified for execution. If we create another listener for graph B with execute graph

task with graph C specified, it will work as a chain of graphs.

179

Chapter 24. Listeners

Figure 24.3. Event source graph isn't specified, thus listener works for all graphs in specified sandbox

180

Chapter 24. Listeners

Email notification about graph failure

Figure 24.4. Web GUI - e-mail notification about graph failure

181

Chapter 24. Listeners

Email notification about graph success

Figure 24.5. Web GUI - email notification about graph success

182

Chapter 24. Listeners

Backup of data processed by graph

Figure 24.6. Web GUI - backup of data processed by graph

183

Chapter 24. Listeners

Jobflow Event Listeners

Jobflow Event Listeners allow you to define a task that the Server will execute as a reaction to the success or failure of executing a specific job (a jobflow).

Each listener is bound to a specific jobflow and is evaluated every time the jobflow is executed (no matter whether manually, through another jobflow, scheduled, via an API call, etc.).

Jobflow Event Listeners work very similarly to Graph Event Listeners ( Graph Event Listeners (p. 177)) in many

ways, since ETL Graphs and Jobflows are both "jobs" from the point of view of the CloverETL Server.

In the Cluster, the event and the associated task are executed on the same node the job was executed on. If the jobflow is distributed, the task will be executed on the master worker node. However, you can override where the task will be executed by explicitly specifying a Node ID in the task definition.

Jobflow Events

Each event carries properties of the event source job. If there is an event listener specified, task may use these properties. E.g. next job in the chain may use "EVENT_FILE_NAME" placeholder which activated the first job in the chain. Job properties, which are set specifically for each run (e.g. RUN_ID), are overridden by the last job.

Types of jobflow events

•

Jobflow started (p. 184)

•

Jobflow phase finished (p. 184)

•

Jobflow finished (p. 184)

•

Jobflow error (p. 184)

•

Jobflow aborted (p. 184)

•

Jobflow timeout (p. 184)

•

Jobflow unknown status (p. 185)

Jobflow started

A Jobflow started event is created, when jobflow execution successfully started.

Jobflow phase finished

A Jobflow phase finished event is created, everytime when jobflow phase is finished and all its nodes are finished with status

FINISHED_OK

Jobflow finished

A Jobflow finished event is created, when all phases and nodes of jobflow are finished with status

FINISHED_OK

Jobflow error

A Jobflow error event is created, when jobflow cannot be executed for any reason, or when any node of the jobflow fails.

Jobflow aborted

A Jobflow aborted event is created, when jobflow is explicitly aborted.

Jobflow timeout

A Jobflow timeout event is created, when jobflow runs longer then specified interval. Thus you have to specify

"Job timeout interval" attribute for each listener of jobflow timeout event. You can specify this interval in seconds or in minutes or in hours.

184

Chapter 24. Listeners

Figure 24.7. Web GUI - jobflow timeout event

Jobflow unknown status

A Jobflow unknown status event is created, when the server, during the startup, detects run records with undefined status in the executions history. Undefined status means, that server has been killed during jobflow run.

Server automatically changes state of jobflow to Not Available and sends jobflow status unknown event.

Please note, that this works just for executions, which have persistent record in executions history. It is possible to execute transformation without persistent record in executions history, typically for better performance of fast running transformations (e.g. using Launch Services).

Listener

User may create a listener for the specified event type and jobflow (or all jobflows in sandbox). The listener is actually connection between jobflow event and task, where jobflow event specifies when and task specifies what to do.

Event handling consist of the following course of actions:

• event is created

• listeners for this event are notified

• each listener performs the related task

Tasks

Task specifies operation which should be performed as the reaction to the triggered event.

Task types are described in Chapter 20, Tasks (p. 144).

Note: You can use a task of any type for a jobflow event listener. Description of task types is divided into two sections just to show the most obvious use cases.

185