TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML User's Guide

TIBCO ActiveMatrix BusinessWorks

™

Plug-in for Large XML

User's Guide

Software Release 6.1.0

January 2016

Two-Second Advantage

®

2

Important Information

SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH

EMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY

(OR PROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THE

EMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY

OTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE.

USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND

CONDITIONS OF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED

SOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE

CLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD

OR INSTALLATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE)

OR IF THERE IS NO SUCH SOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USER

LICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE “LICENSE” FILE(S) OF THE

SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, AND

YOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BE

BOUND BY THE SAME.

This document contains confidential information that is subject to U.S. and international copyright laws and treaties. No part of this document may be reproduced in any form without the written authorization of TIBCO Software Inc.

TIBCO, Two-Second Advantage, and TIBCO ActiveMatrix BusinessWorks are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries.

Enterprise Java Beans (EJB), Java Platform Enterprise Edition (Java EE), Java 2 Platform Enterprise

Edition (J2EE), and all Java-based trademarks and logos are trademarks or registered trademarks of

Oracle Corporation in the U.S. and other countries.

All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.

THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOT

ALL OPERATING SYSTEM PLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASED

AT THE SAME TIME. SEE THE README FILE FOR THE AVAILABILITY OF THIS SOFTWARE

VERSION ON A SPECIFIC OPERATING SYSTEM PLATFORM.

THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER

EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF

MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT.

THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL

ERRORS. CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE

CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO

SOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S)

AND/OR THE PROGRAM(S) DESCRIBED IN THIS DOCUMENT AT ANY TIME.

THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY OR

INDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE,

INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES.

Copyright

©

2013-2016 TIBCO Software Inc. All rights reserved.

TIBCO Software Inc. Confidential Information

TIBCO ActiveMatrix BusinessWorks

™

Plug-in for Large XML User's Guide

Contents

TIBCO Documentation and Support Services

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4

Accessing Documentation and Changing Help Preferences

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5

Introduction

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

Overview

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

The Large XML Palette

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

Activities

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

Input Sources

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

FileToStream Activity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

GetFragment Activity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10

StreamToFile Activity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

TransformXML Activity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

XMLSplitter Activity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

XMLValidation Activity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

File Reusability in Plug-in Activities

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23

Limitations

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

XML Split Elements

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

Example

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

Example - Creating Fragments with Required Parents Only

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

Example - Split Element is a Subelement

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

Samples

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

Sample of GetFragment Activity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

Sample of GetFragment Activity Using Dynamic Input Source

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29

Sample of GetFragment Activity Using TCP Stream

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

Sample of TransformXML Activity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

Sample of XMLSplitter Activity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

Sample of XMLValidation Activity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

33

3


™


TIBCO Documentation and Support Services

Documentation for this and other TIBCO products is available on the TIBCO Documentation site. This site is updated more frequently than any documentation that might be included with the product. To ensure that you are accessing the latest available help topics, please visit: https://docs.tibco.com

●

●

●

Product-Specific Documentation

The following documents for this product can be found on the TIBCO Documentation site:

TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML Installation

TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML User's Guide

TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML Release Notes

How to Contact TIBCO Support

For comments or problems with this manual or the software it addresses, contact TIBCO Support:

●

For an overview of TIBCO Support, and information about getting started with TIBCO Support, visit this site: http://www.tibco.com/services/support

●

If you already have a valid maintenance or support contract, visit this site: https://support.tibco.com

Entry to this site requires a user name and password. If you do not have a user name, you can request one.

How to Join TIBCOmmunity

TIBCOmmunity is an online destination for TIBCO customers, partners, and resident experts. It is a place to share and access the collective experience of the TIBCO community. TIBCOmmunity offers forums, blogs, and access to a variety of resources. To register, go to the following web address: https://www.tibcommunity.com

4


™


Accessing Documentation and Changing Help Preferences

By default, access to the TIBCO ActiveMatrix BusinessWorks

™

Plug-in for Large XML documentation from TIBCO Business Studio, is through https://docs.tibco.com. However, if you do not have an all time

Internet access, you can download the complete TIBCO ActiveMatrix BusinessWorks Plug-in for Large

XML 6.X.X documentation set and configure TIBCO Business Studio to this local copy.

Accessing Documentation

The TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML documentation can be accessed in the following ways:

●

●

Online Mode: The online documentation is available on the TIBCO Product Documentation site at https://docs.tibco.com/products/tibco-activematrix-businessworks-plug-in-for-large-xml . The Doc site contains the latest version of documentation. Check the Doc site frequently for the updated documentation.

Offline Mode: For environments that do not have Internet access, the product documentation can be downloaded from the TIBCO Product Documentation site using the Download All button, or from https://download.tibco.com/tibco/ , and made available offline either on a local or a network drive.

By default, TIBCO Business Studio accesses the documentation from the TIBCO Product

Documentation site. To access the product documentation using the offline mode, change the help preferences in TIBCO Business Studio to access help from a local directory or from an internal web server.

Prerequisites

Before changing the help preferences, download TIBCO ActiveMatrix BusinessWorks Plug-in for Large

XML 6.X.X documentation from https://docs.tibco.com/ .

1. Click https://docs.tibco.com/ .

2. In the Search field, type

ActiveMatrix BusinessWorks Plug-in for Large XML

and press

Return.

3. Select TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML 6.X.X from the list.

4. Scroll to the bottom of the page and click the Download All button.

5. A zip file containing the latest documentation downloads to your web browser's default download location. Copy the zip file to a local directory, or to an internal web server and then unzip the file.

Changing Help Preferences

Procedure

1. In TIBCO Business Studio, click Window > Preferences. On Mac OS X, click TIBCO Business

Studio > Preferences.

2. In the Preferences dialog, click BusinessWorks > Help.

3. Under BusinessWorks Large XML Plugin, click Custom Location and then click Browse to select the html directory where you unzipped the documentation, or provide the URL to the html directory on your internal web server.

4. Click Apply and then click OK.

5


™


Introduction

This section introduces you to TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML.

Overview

TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML can be used to process large XML files.

Processing of Large XML documents is a critical use case for most integration projects. It needs the processing capability to be optimized in terms of memory and CPU usage to achieve the required throughput and performance. The current ActiveMatrix BusinessWorks XML palette provides basic

XML processing where XML is always loaded in memory. This poses significant challenges and limitations for processing the large XML documents.

TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML addresses the large XML processing issue for most of the use cases. The key to large XML processing is Streaming where the large XML is never loaded into memory. It provides the following high level capabilities:

●

●

●

The user can split a stream of large XML into fragments based on configurable criteria and, then, process the fragments without bloating the memory and also provide smart recovery options.

The user can validate a stream of large XML against a configured schema without loading the large

XML into memory.

The user can transform a stream of large XML by configuring an XSLT.

The user can read and write streams from or to the file system.

The XML processing using the plug-in is as follows:

6


™


1. Input Large XML: You can source large XML from an already created stream or directly creating a stream from File.

2. XML Validation (optional): You can, optionally, validate the large XML Stream against a schema.

Memory and CPU usage is optimum as the large XML is not loaded into memory.

3. XML Transformation (optional): You can, optionally, transform the large XML based on a configured XSLT. Memory and CPU utilization is optimum as compared to the out-of-the-box XML processing in TIBCO ActiveMatrix BusinessWorks.

4. XML Processing: You can configure the criteria to split the XML by providing the split element configuration in addition to either number of records or size of fragment. You can specify the destination location where all the XML fragments can be written to a file using the XMLSplitter activity.

Alternately, you can use the GetFragment activity in a group to process every XML fragment.

●

●

●

During fragment processing, you get additional meta data about each fragment being created which helps co-relate the fragment with the large XML.

The earlier fragment is auto-cleaned from memory before the next fragment is output by the

GetFragment activity running in a loop. This guarantees that only the fragment getting processed is in memory at any given time.

It also provides smart recovery mechanism by which you can add the Checkpoint activity in the group while processing a fragment of the large XML.

5. Output: You can output XML as a string for further processing or can write the string directly to a file.

6. Splitter Output: You can output the total number of fragments created and the location where the split files are stored.

The Large XML Palette

The Large XML palette provides the following activities:

●

●

●

●

●

●

FiletoStream : reads a file with XML content and outputs a stream for downstream processing.

GetFragment : used in a Group to get a fragment of the large XML based on the configuration for

further processing.

StreamToFile

: writes stream of XML to a file.

TransformXML

: provides streaming transformation of XML based on the configured XSLT.

XMLSplitter : is a utility to split large XML into fragments and write it out without processing each

fragment.

XMLValidation

: provides streaming validation of Large XML files against the configured XSD.

7


™


Activities

This section provides information about the activities in the Large XML palette.

Input Sources

This section provides information on the TCP Stream and Dynamic Input Stream input sources.

TCP Stream

A TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML activity can accept a TCP stream as an input XML source. Currently, only the GetFragment activity supports a TCP stream as an input XML source. For this, you require:

●

●

A TCP plug-in activity, such as TCP Open Connection, which is used to create and open a connection with the TCP server providing the large XML TCP stream. This connection is used in the

GetFragment activity as input. For more information on the TCP plug-in activities, refer to the "TCP

Connection" section of TIBCO ActiveMatrix BusinessWorks Palette Reference.

The TCP server providing the large XML TCP stream must be running.

A sample process is as follows:

8

You can specify a timeout period, that is, the time to wait (in milliseconds) for a TCP stream over a TCP connection. If the timeout expires, a Read timeout error is thrown. If the connection is closed, a

Connection Reset error is thrown.

TCP Stream as Input and Output

●

●

Reading from a TCP stream: The TCP stream is treated as input and processed depending upon the

GetFragment activity's configuration.

Writing to a TCP stream: The output can be directly mapped to TCP activities such as TCP Write, as the GetFragment activity supports binary type of output.

Dynamic Input Source

With a Dynamic Input Source, a TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML activity can get fragments from a file or input stream while the file or input stream is being written to. That is, the file or input stream, is incomplete.

Currently, only the GetFragment activity supports a dynamic input source.


™


9

FileToStream Activity

The FileToStream activity creates an input stream of a file from the disk. The file is not loaded in the memory, instead the activity opens a stream and passes on the input stream Object Reference in the output to be used in other activities.

General Tab

The General tab has the following fields.

Field

Name

Literal Value/ Module Property/

Process Property?

No

Description No

Description

The name to be displayed in the process definition

A short description of the activity

Input Tab

The input for the activity is the following.

Input Item

fileName

Datatype

String

Output Tab

The output of the activity is the following.

Description

The path and name of file

Output Item

inputStream

Datatype

Object

Reference

Description

The input stream reference to be used for further processing

Fault Tab

The Fault tab lists the possible exceptions that can be thrown by this activity. See TIBCO ActiveMatrix

BusinessWorks Error Codes for more information about the error codes and corrective actions to take.

Exception

FileNotFoundException

Thrown When...

Cannot find the file provided in the fileName parameter


™


10

Sample

The

TIBCO_HOME/bw/palettes/lx/<version>/samples

folder contains samples and a readme file that explains how to import and use the sample.

GetFragment Activity

●

●

●

The GetFragment activity provides a way to process a large stream of XML fragment-by-fragment, based on a configuration.

This activity must be used in a group. It outputs next fragment each time the group executes.

For more information on how the GetFragment activity fragments the large XML, refer to XML Split

Elements .

The GetFragment activity also provides the following features:

●

Memory Optimization: When the GetFragment activity is used in a loop, it is ensured that only the fragment being processed is loaded into memory. The earlier ones are cleared from memory.

Smart recovery: Recovery can be done using the Checkpoint activity of TIBCO ActiveMatrix

BusinessWorks. In case the engine crashes during processing in a group, on re-starting, the processing would start from the Checkpoint activity from the fragment where it stopped.

The recovery is not supported if GetFragment activity Input Source is selected as input

stream.

For more information on the Checkpoint activity, refer to TIBCO ActiveMatrix BusinessWorks Palette

Reference.

Metadata Information: This activity outputs metadata information such as fragment number, number of records, and total number of fragments processed for every fragment generated. This metadata is useful to corelate the fragments with each other and with the parent large XML.

This metadata information is not provided by the XMLSplitter activity.

In-line Fragment Processing: Processes each XML fragment in-line while the XML is being fragmented and still guarantees only one fragment in memory at any time.

With the XMLSplitter

activity, you need to wait for the activity to complete before you can apply your business logic.

General Tab



™


Field

Name

Description

Input XML Source

(Mandatory)

Dynamic Input

Source (Optional)

Timeout(msec)

Output Source

(Mandatory)

No

No

No

Literal

Value/

Module

Property/

Process

Property?

Description



●

●

●

●

The format of the input XML: file name input stream output stream tcp stream

No

Yes

No

This check box is enabled only when the Input XML Source is selected as file name or input stream.

With a Dynamic Input Source, the GetFragment activity can get fragments from a file or input stream while the file or input stream is being written to. That is, the file or input stream, is incomplete.

You can specify the following values:

●

●

Read Timeout for Dynamic Source (Msec): The time to wait, in milliseconds, for a dynamic stream. The default value is 10000.

Retry Count: The number of times the Read Timeout for

Dynamic Source (Msec) should be retried before an error or exception is thrown. The default is 1.

The Read Timeout for Dynamic Source (Msec) and Retry

Count must to be adjusted to ensure that the dynamic stream is handled correctly.

This field is enabled only when the Input XML Source is selected as tcp stream.

The time to wait, in milliseconds, for a TCP stream over a TCP connection.

●

●

●

The format of the output: output stream xml binary

11


™


Field

Fragment Criteria

(Mandatory)

Fragment Criteria

Value

Literal

Value/

Module

Property/

Process

Property?

Description

No

●

●

The criteria used for fragmenting: size of fragment - split size in bytes number of records

Yes The value of Fragment Criteria.

Note: You can also specify the fragment criteria in the Input tab (fragmentSize and noOfRecords). The fragment criteria specified in the Input tab overrides the Fragment Criteria specified in the General tab.

Input Tab


Input Item

fileName

Datatype

String inputStream outputStream elementName

ObjectReference

ObjectReference

String

Description

The path and name of XML file.

This field is available only when file name is selected as the Input XML Source on the General tab.

The stream mapped from output of FileToStream activity.

This field is available only when input stream is selected as the Input XML Source on the General tab.

The Output stream ObjectReference mapped from the

GetFragment activity.

It is available only when Output Stream is selected as the Input XML source on the General tab.

The name of the XML element to be used for splitting the input XML document into fragments.

If you want to consider parent elements during fragmenting, specify the split element name as an expression. For example:

/root/parent/split element

NOTE: The path must begin with a forward slash (/).

For more information on how the GetFragment

activity fragments the large XML, refer to XML Split

Elements .

12


™


13

Input Item

fragmentSize noOfRecords

Datatype

Integer

Integer

Description

Specifies the maximum size of the fragment in bytes of each fragment.

Available only if size of fragment is selected as the

Fragment Criteria on the General tab.

Note: The fragment criteria value specified in the

Input tab overrides the Fragment Criteria Value specified in the General tab.

Specifies the maximum number of records in each fragment.

Available only if number of records is selected as the

Fragment Criteria on the General tab.

Note: The Fragment Criteria Value specified in the

Input tab overrides the Fragment Criteria Value specified in the General tab.

Output Tab


Output Item

fragmentOutputStream

Datatype

Object

Reference

XML binaryContent endofFile

String

Binary

Boolean

Description

The object reference of stream which can be used with StreamToFile, TransformXML, and

XMLValidation activities.

Available if output stream is selected as the

Output Source on the General tab.

Available when xml is selected in the Output

Source field on the General tab. This item contains the fragmented XML string.

The fragment content in binary format.

Available when binary is selected as the Output

Source on the General tab.

This field shows whether the GetFragment activity has finished processing the given XML file or XML stream. If the end of the file has reached, the value is true

. Otherwise, the value is false

.

This can be used to put a condition in the Loop

Group activity to run the iteration till the condition is not true

.

For example: true()=$GetFragment/endOfFile or

$GetFragment/endOfFile=true()


™


14

Output Item

fragmentNumber noOfrecords

Datatype

Integer

Integer

Description

The sequential number of the currently generated fragment

The number of records contained in the current fragment

The total number of records processed totalProcessedFragments integer

Fault Tab



Exception

GetFragmentException


FragmentCriteriaException

Thrown When...

This is a Generic GetFragment activity exception. This exception provides errors related to fragment processing.

Cannot find the file provided in the fileName parameter.

This exception provides errors related to fragment criteria.

The excessSplitElement in FragmentCriteriaException holds the excess split element in binary form. You can handle these errors to process the excess split element separately and still continue splitting the large XML file.

Sample

The



StreamToFile Activity

The StreamToFile activity writes a file to the disk from the output data stream.

General Tab



™


Field

Name

Description

Create Non-Existing

Directories (Optional)

No

No

No

Literal Value/

Module

Property/

Process

Property?

Description



When this check box is selected, the activity creates all directories in the specified path, if they do not already exist.

If this check box is not selected and one or more directories in the specified path do not exist, it throws an exception.

Input Tab


Input Item

outputStream fileName

Datatype

Object

Reference

String

Description

The output stream mapped from activities such as

GetFragment and XMLTransform

The path and the name of file where the output stream data is to be written

Output Tab


Output Item

fileName

Datatype

String

Description

The complete path where the file is written on the disk.

By default, any existing file is over-written. It cannot be appended.

Fault Tab



Exception


FileIOException

StreamToFileException

Thrown When...

Cannot find the file provided in the filename parameter.

No such file or directory exists for the output file name.

An invalid output stream is mapped as input to the activity.

15


™


16

Properties

To set the number of threads used to create the fragment files, set the bw.plugin.largeXml.streamWriter.maxThreads

plug-in property. The default is 10 threads.

Sample

The



TransformXML Activity

The TransformXML activity transforms large XML files using the specified XSLT.

General Tab


Field

Name

Description

Input XML Source

(Mandatory)

Output Source

(Mandatory)

Stylesheet

No

No

No

Literal Value/

Module

Property/

Process

Property?

Description



●

●

●

The format of the input XML: file name input stream output stream

No

●

●

●

The format of the output: output stream xml binary

No The stylesheet (XSLT) to be used to transform the large

XML file.

Note: You can also specify a stylesheet file name in the

Input tab. The stylesheet specified in the Input tab overrides the stylesheet specified in the General tab.


™


Input Tab


Input Item

fileName

InputStream outputStream stylesheet

Datatype

String

ObjectReference

ObjectReference

String

Description

The path and name of XML file to be transformed.

This field is available only when File

Name is selected as the Input XML

source.

Object reference of the Input Stream mapped from the output of FileToStream activity.

This field is available only when input

stream is selected as the Input XML

source.

The Output stream ObjectReference mapped from the GetFragment activity.

It is available only when Output Stream is selected as the Input XML source on the General tab.

Full path of stylesheet which is used to transform the XML file.

Note: The stylesheet specified in the

Input tab overrides the stylesheet specified in the General tab.

Output Tab


Output Item

outputStream

Datatype

Object

Reference

XML String binaryContent Binary

Content

Description

The output string reference of the transformed XML which can be used with the StreamToFile and

XMLValidation activities.

Available when xml is selected in the Output Source field on the General tab. This item contains the transformed XML string.

Available when binary is selected in the Output

Source on the General tab.

The transformed XML content in binary format.

17


™


18

Fault Tab



Exception

TransformXMLException


Thrown When...

This is a generic Transform activity exception. This exception provides errors related to XML transformation.


Sample

The



XMLSplitter Activity

The XMLSplitter activity splits a large XML file into multiple fragments depending on the specified fragment criteria and outputs it to the configured location without allowing any processing of fragments in the ActiveMatrix BusinesssWorks process.

For more information on how the XMLSplitter activity splits the large XML, refer to

XML Split

Elements .

General Tab


Field

Name

Description

Input XML Source

(Mandatory)

No

No

No

Literal Value/

Module

Property/

Process

Property?

Description



●

●

●

The format of the input XML: file name input stream output stream


™


19

Field

Fragment Criteria

(Mandatory)

Fragment criteria value

(Optional)

Create Non-Existing

Directories (Optional)

Literal Value/

Module

Property/

Process

Property?

No

Description

●

●

The criteria used for fragmenting: size of fragment - split size in bytes number of records

Yes

No

The value of Fragment Criteria.

Note: You can also specify the fragment criteria value in the Input tab. The fragment criteria value specified in the

Input tab overrides the fragment criteria value specified in the General tab.

When this check box is selected, the activity creates all directories in the specified path, if they do not already exist.

If this check box is not selected and one or more directories in the specified path do not exist, it throws an exception.

Input Tab


Input Item

fileName

Datatype

String inputStream outputStream

ObjectReference

ObjectReference

Description

Complete path of the XML file to split.

It is available only when File is selected as the Input XML Source on the

General tab.

Object reference of the Input Stream or

Output Stream mapped from the output of FileToStream or GetFragment activity.

It is available only when Input Stream is selected as the Input XML Source on the General tab.


It is available only when Output

Stream is selected as the Input XML

source on the General tab.


™


20

Input Item

elementName fragmentSize fragmentPrependName noOfRecords fragmentStoreLocation

Datatype

String

Integer

String

Integer

String

Description

Name of the split element to be used for splitting the input XML document into smaller files.

If you want to consider parent elements during fragmenting, specify the split element name as an expression. For example:

/root/parent/split element

The path must begin with a forward slash (/).

For more information on how the

XMLSplitter activity splits the large

XML, refer to

XML Split Elements

.

Maximum size of the fragment (in bytes) to be used for fragmenting

Prefix to be added to the file name when creating the fragment name.

If fragmentPrependName is not set, the default fragment name used is

"

XMLFragment-

".

If fragmentPrependName is set, the fragment name is created as fragmentPrependName

+numberoffragment+". xml"

. For example, splitterOutput-1.xml

, splitterOuput-2.xml

, splitterOuput-3.xml

, and so on.

Maximum number of records to be used for fragmenting

Location where the split files are to be stored

Output Tab


Output Item

totalFragments fragmentStoreLocation

Datatype

Integer

String

Description

Total number of fragments created

Location where the split files are stored


™


21

Fault Tab



Exception

XMLSplitterException


FragmentCriteriaException

Thrown When...

This is a Generic Splitter Activity Exception. This Exception provides errors related to fragment processing.


This exception provides errors related to fragment criteria.

The excessSplitElement in FragmentCriteriaException holds the excess split element in binary form. You can handle these errors to process the excess split elements separately.

Properties

To set the number of threads used to create the fragment files, set the bw.plugin.largeXml.splitter.maxThreads

plug-in property. The default is 10 threads.

Sample

The



XMLValidation Activity

The XMLValidation activity validates whether an XML document is an instance of the specified schema. The activity validates the XML file against an XSD without loading it into memory.

General Tab


Field

Name

Description

Literal Value/

Module

Property/

Process

Property?

No

No

Description




™


Field

Input XML Source

(Mandatory)

Schema

Literal Value/

Module

Property/

Process

Property?

No

No

Description

●

●

●

●

The format of the input XML source: file name input stream output stream binary

The schema file to be used when validating the XML file.

Note: You can also specify a schema file name in the

Input tab, if required. The schema specified in the

Input tab overrides the schema specified in the

General tab.

Input Tab


Input Item

fileName

Datatype

String

Input Stream outputStream

Binary

ObjectReference

ObjectReference

Bytes

Description

The path and name of the file.

This field is available only when File Name is selected as the Input XML source on the

General tab.

Input Stream ObjectReference mapped from the output of the FileToStream activity.

It is available only when Input Stream is selected as the Input XML source on the

General tab.


It is available only when Output Stream is selected as the Input XML source on the

General tab.

XML data in binary format to validate.

It is available only when Binary is selected as the Input XML source on the General tab.

22


™


23

Input Item

schemaFileName

(Optional)

Datatype

String

Description

The location and filename of the schema file to be used when validating the XML file.

Note: The schema specified in the Input tab overrides the schema specified in the

General tab.

Fault Tab



Exception

FileIOException


XMLSchemaValidationException

Thrown When...

This error may be thrown when an XML file is being read from a stream.

Cannot find the file provided in the fileName parameter or schemaFileName parameter.

This is a generic validation exception. This exception provides errors related to XML validation.

Sample

The



File Reusability in Plug-in Activities

The file reusability scenarios for stream and file handling are listed here.

●

When the FileToStream activity is used in a process, the file used in the FileToStream activity can be reused only when it is successfully processed by the following activities. If these activities throw an exception while using the input stream created from the FileToStream activity, the stream is closed to ensure that the file can be reused.

●

—

—

—

—

—

—

—

—

GetFragment

XMLSplitter

XMLValidation

TransformXML

When a large XML file name is directly used as input source with the following activities, the file can be reused when these activities complete successfully or if they throw an error.

GetFragment

XMLSplitter

XMLValidation

TransformXML


™


24

●

●

If the FileToStream activity is used in a process followed by any TIBCO ActiveMatrix

BusinessWorks activity (non-BWLX plug-in activity) and an error is thrown, the file used with the

FileToStream activity is released when the process instance ends.

When FragmentCriteriaException is thrown, the GetFragment activity can still iterate through the fragment by skipping the excessSplitElement given out in the FragmentCriteriaException. To allow this functionality, the file name and input stream handles are not reused if you do not iterate back to the GetFragment activity on error handling. In this case, the file handle is released when the job ends.

Limitations

●

●

If a BusinessWorks activity such as Generate Error is used in the same group as the GetFragment activity, the related file is locked until the job ends.

The GetFragment activity carries excess split elements in memory to be written with the next fragment. The activity might carry all the excess size split elements till the end of fragment processing but it does not output the excess split elements after the processing is completed. You can create a separate error or output stream for such split elements, process them separately, and still continue splitting the large XML file.


™


XML Split Elements

This section explains how the GetFragment and XMLSplitter activities fragments the large XML files.

XML fragments are created based on the split element. The split element name (elementName) is specified in the Input tab of the GetFragment and XMLSplitter activities.

Example

An

Employees

XML file has employee records with all the details of employees.

In the fragment XML criteria, the

<Employee>

element name is used to split the original XML into smaller fragments.

Original XML

<?xml version="1.0" encoding="ISO-8859-1"?>

<Employees>

<Employee>

<name>Josephine Foller</name>

<department>Marketing</department>

<country>USA</country>

<employ-type>Permanent</employ-type>

<year>1985</year>

</Employee>

<Employee>

.............

.............

</Employee>

</Employees>

Fragment XML


<Employees>

<Employee>





<year>1985</year>

</Employee>

</Employees>

Example - Creating Fragments with Required Parents Only

In this example, the split element is a subelement, <country>, which is a part of the <Employee> element.


<Employees>

<Employee>





<year>1985</year>

</Employee>

<Employee>

<name>Kris Amigon</name>


<country>UK</country>


<year>1997</year>

</Employee>

<Employee>

.............

.............

</Employees>

25


™


26

The fragments created contain one fixed header which includes unrequired elements such as <name> and <department> element.


<Employees>

<Employee>





<year>1985</year>

</Employee>

<Employee>





<year>1997</year>

</Employee>

<Employee>

.............

.............

</Employees>


<Employees>

<Employee>

<name>Josephine Foller</name>

<department>Marketing</department>


.............


</Employee>

</Employees>

If you want the fragment to contain only the repeating subelements (without the above unrequired elements), you can specify the split element as a /Employees/Employee/country expression in the Input tab of the activity.


<Employees>

<Employee>


.............


</Employee>

</Employees>

Example - Split Element is a Subelement

In this example, the split element is a subelement, <country>, which is a part of the <Employee> element.


<Employees>

<Employee>





<year>1985</year>

</Employee>

<Employee>





<year>1997</year>

</Employee>

<Employee>

.............


™


27

●

●

.............

<Dept>

<name>Marketing</name>


</Dept>

</Employees>

The fragments created contain the <country> element from the <Employee> element as well as the

<Dept> element.


<Employees>

<Employee>


<department>marketing</department>

<country>usa</country>

.............

<country>UK</country>

</Employee>

</Employees>

If you want the fragment to contain only the repeating elements from the <Employee> element, use two

GetFragment activities.


<Employees>

<Employee>


.............

</Employee>

</Employees>

Use two GetFragment activities as follows:

The first GetFragment activity splits the XML with the <Employee> element.

The output of the first activity is mapped to the second GetFragment activity input where the split element name is specified as /Employees/Employee/country.


™


28

Samples

You can run a sample end-to-end and understand how the TIBCO ActiveMatrix BusinessWorks Plug-in for Large XML works.

Sample of GetFragment Activity

The GetFragment activity is designed to iterate through the XML fragments one by one. This activity is designed to run with the Loop Group activity to allow iterating through all the XML fragments. In this sample, the GetFragment activity splits the XML file into fragments depending on Fragment Criteria.

The file names used in this sample are specified in module property.

Prerequisites

Before executing the project, set the module property

LargeXML_Root

to the location of the plug-in samples

folder. For example, if the plug-in samples are available in

$TIBCO_HOME/bw/palettes/lx/

<version>/samples

, set

LargeXML_Root

to $


. The rest of the input and output paths are configured using this module property.

Procedure

1. In the samples directory, select <TIBCO HOME> bw > palettes > lx > <version> > samples and double-click tibco.bw.sample.palette.bwlx.LargeXMLPluginSample

.

2. In Project Explorer, expand the tibco.bw.sample.palette.bwlx.LargeXMLPluginSample

project.

3. Set the default ApplicationProfile to match your operating system.

4. Fully expand the

Processes

directory and double-click name

XMLGetFragment.bwp

. In Module

Descriptors > Components, you can see

XMLGetFragment.bwp

.

5. Click Run > Debug Configurations.

6. In the left tree of the Debug Configuration wizard, expand BusinessWorks Application and select

BWApplication.

7. Click the Applications tab and then click Deselect All if you have multiple applications. Select the check box next to

tibco.bw.sample.palette.bwlx.LargeXMLPluginSample.application

.

8. Click Debug. This runs the sample in the Debug mode.

9. Click the Terminate icon to stop the process.

Result

●

●

The number of split files created as per Fragment Criteria in


<version>/samples/LargeXML/data/output/

.

End of file (EOF) must be true at the end of the iteration.

●

●

●

Understanding the Configuration

The following activities are used in this sample:

FileToStream: The activity creates an input stream of a file from the disk.

GetFragment: Splits the XML file in loop group activity according to Fragment Criteria.

StreamToFile: The activity writes a file to the disk from the output data stream.


™


29

Sample of GetFragment Activity Using Dynamic Input Source

The GetFragment activity is designed to iterate through the XML fragments one by one. This activity is designed to run with Loop Group activity to allow iterating through all the XML fragments. In this sample, the GetFragment activity accepts a dynamic input source that is, a live stream as an input and splits the XML file into fragments depending on the Fragment Criteria. The file names used in this sample are specified in module property.

Prerequisites


LargeXML_Root




<version>/samples

, set

LargeXML_Root

to $



Procedure

1. In the samples directory, select <TIBCO HOME > bw > palettes > lx > <version> > samples and double-click tibco.bw.sample.palette.bwlx.LargeXMLPluginSample

.


project.


4. Fully expand the Processes directory and double-click the name

GetFragmentUsingDynamicInputstream.bwp

and

LivestreamWriterSimulator.bwp

. In Module


GetFragmentUsingDynamicInputstream.bwp

and

LivestreamWriterSimulator.bwp

.

5. Click Run > Debug Configurations.

6. In the left tree of the Debug Configuration wizard, expand BusinessWorks Application, and select

BWApplication.

7. Click the Applications tab and then click Deselect All if you have multiple applications. Select the check box next to tibco.bw.sample.palette.bwlx.LargeXMLPluginSample.application

.



Result

●

●




.



™


30

●

●



●

●



There are two processes, a Livestream Writer Simulator and a GetFragment Using Dynamic

Inputstream.

The Livestream Writer Simulator generates a live stream which is an input to GetFragment activity.

In GetFragment Using Dynamic Inputstream, the following options are set:

—

—

—

—

Input XML Source is selected as input stream

Dynamic Input Source check box is enabled

timeout is set to 1000msec

Output Source is set to output stream

The Fragment Criteria is set to no. of records. The Fragment Criteria Value is the value required for splitting the XML file according to Fragment Criteria. The elementName is the element used as split criteria. The files are written to the disk using StreamToFile activity at a specified location.

Sample of GetFragment Activity Using TCP Stream

The GetFragment activity is designed to iterate through the XML fragments one by one. This activity is designed to run with Loop Group activity to allow iterating through all the XML fragments. The

GetFragment activity accepts a TCP stream as an input and splits the XML file into fragments depending on the fragment criteria specified. The file names used in this sample are specified in module property.

Prerequisites


LargeXML_Root




<version>/samples

, set

LargeXML_Root

to $



Procedure


.


project.


4. Fully expand the Processes directory and double-click

GetFragmentUsingTCPStream.bwp

and

TCPServerSimulator.bwp

. In Module Descriptors > Components, you can see

GetFragmentUsingTCPStream.bwp

and

TCPServerSimulator.bwp

.



BWApplication.


™


31


.



Result

●

●




.




●

●



There are two processes, TCP Server Simulator and GetFragment Using TCP Stream, for splitting elements.

●

●

The TCP Server Simulator process generates a TCP stream which is used as an input to the

GetFragment activity for splitting it into chunks. The input and output paths are configured using global variables.

The GetFragment Using TCP Stream process, opens a TCP connection, selects input stream as the Input XML Source, and Output Source is set to output stream.

By default, Fragment Criteria is the size of the fragment. The Fragment Criteria Value is the value required for splitting the XML file according to Fragment Criteria. The elementName is the element used as split criteria. The files are written to the disk at specified location.

Sample of TransformXML Activity

This sample highlights the use of the TransformXML activity to transform an input XML document into the required output format. In this sample, the XML file is transformed to an HTML format using

XSLT. The file names used in this sample are specified in module property.

Prerequisites

1. Import the XSLT file to the schemas

folder. Right-click on schemas > import > file system > browse to the data folder and select

$TIBCO_HOME/bw/palettes/lx/<version>/samples/LargeXML/ data/input/listemployee.xslt.

2. Set the XSLT in the Stylesheet field of the General tab.

3. Before executing the project, set the module property

LargeXML_Root



$TIBCO_HOME/bw/ palettes/lx/<version>/samples

, set

LargeXML_Root

to $

TIBCO_HOME/bw/palettes/lx/

<version>/samples


Procedure


.


™


32


project.


4. Fully expand the

Processes

directory and double-click

TransformXML.bwp

. In Module Descriptors

> Components, you can see

TransformXML.bwp

.



BWApplication.


.



Result

An HTML file is generated in the output

folder by the name bwlx_TransformOutputFile.html

.

●

●

●



FileToStream: The activity creates an input stream of a file from the disk.

TransformXML: The activity transforms an XML file to HTML format.


Sample of XMLSplitter Activity

This sample highlights the use of XMLSplitter activity. It accepts input XML file as file name. It splits the XML file according to the Fragment criteria, that is, Size of Fragment. This activity writes all the fragments to the disk and at the output, it provides the total number of fragments generated and the fragment store location path where fragments are written to the disk. The file names used in this sample are specified in module property.

Prerequisites


LargeXML_Root




<version>/samples

, set

LargeXML_Root

to $



Procedure


.


project.


4. Fully expand the

Processes


XMLSplitter.bwp

. In Module Descriptors >

Components, you can see

XMLSplitter.bwp

.



BWApplication.


™


33


.



Result

●




.

●


The following activity is used in this sample:

XMLSplitter: The activity splits a large XML file into multiple fragments depending on the specified fragment criteria and outputs it to the configured location without allowing any processing of fragments in the Active Matrix BusinessWorks process.

Sample of XMLValidation Activity

This sample highlights the use of XMLValidation activity. The XML file is validated against schema.

The file is valid only if the XML file is an instance of XML schema. Otherwise, the activity throws an error with the details of validation error. The file names used in this sample are specified in module property.

Prerequisites

1. Import the schema file to the schemas

folder. To do this, right-click on schemas > import > file system > browse to the data folder and select

$TIBCO_HOME/bw/palettes/lx/<version>/ samples/LargeXML/data/input/BW_shipping_costs.xsd

.

2. Set the XSD in the Schema field of the General tab.

3. Before executing the project, set the module property

LargeXML_Root



$TIBCO_HOME/bw/ palettes/lx/<version>/samples

, set

LargeXML_Root

to $

TIBCO_HOME/bw/palettes/lx/

<version>/samples


Procedure

1. In the samples directory, select <TIBCO HOME> bw > palettes > lx > <version> > samples and double-click

tibco.bw.sample.palette.bwlx.LargeXMLPluginSample

.


project.


4. Fully expand the

Processes


XMLValidation.bwp

. In Module


XMLValidation.bwp

.



BWApplication.


.


™




Result

The Console displays the output as:

21:13:59.535 INFO [Job_Executor0] com.tibco.thor.frwk.Application - TIBCO-THOR-FRWK-300021: All Application dependencies are resolved for Application

[tibco.bw.sample.palette.bwlx.LargeXMLPluginSample.application:1.0]

21:14:18.383 INFO [Job_Executor0] com.tibco.thor.frwk.Application - TIBCO-THOR-FRWK-300006: Started BW Application

[tibco.bw.sample.palette.bwlx.LargeXMLPluginSample.application:1.0]

21:14:18.568 INFO [bwThread:In-Memory STWorkProcessor-1] c.t.b.p.g.L.t.b.s.p.b.L.Valid-

-----------Validation Successful------------


The following activity is used in this sample:

●

XMLValidation: The activity validates XML files against XSD.

34


™


No results