SAP HANA Smart Data Streaming Developer Guide

Inputs and Outputs [page 26] for details)

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

2.3.1 Streams

Streams subscribe to incoming events and process the event data according to the rules you specify (which you can think of as a "continuous query") to publish output events. Because they are stateless, they cannot retain data, and they use little memory because they do not store events.

Streams can be designated as input or derived. Input streams are the point at which data enters the project from external sources via adapters. A project may have any number of input streams. Input streams do not have continuous queries attached to them, although you can define filters for them. Derived streams either direct data locally within a project, or publish it to external destinations.

Because a stream does not have an underlying store, the only thing it can do with arriving input events is to insert them. Insert, update, and upsert opcodes are all treated as inserts. Delete and safedelete are ignored.

The only opcode that a stream can include in output event records is insert.

If you specify a key on a stream, the stream's opcode handling semantics change, and it becomes a keyed stream. A keyed stream can pass insert, update, and delete events with the opcode unchanged, but will treat them all the same.

Local and output streams take their input from other streams or windows, rather than from adapters, and they apply a continuous query to produce their output. Local streams are identical to output streams, except that local streams are hidden from outside subscribers. Thus, a subscriber cannot subscribe to a local stream. You cannot monitor or subscribe to local streams in the smart data streaming plugin for SAP HANA studio.

Each subscribed stream has a single thread that posts rows to all clients. If one subscriber to this stream backs up and the client's queue is filled, it blocks subscription for all other clients.

In this section:

Keyed Streams [page 25]

Keyed streams save resources by letting you pass insert, update, and delete events through a project without storing the events in memory. Keyed streams also let your project perform certain relational operations, including joins, computes, and filters, without storing the data in memory.

Related Information

Windows [page 28]

2.3.1.1 Keyed Streams

To create a keyed stream, define a primary key for a stream. Inserts, updates, and deletes are assumed to be related to this primary key.

Like other streams, keyed streams can be either input or derived.

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

P U B L I C

A keyed stream:

● Supports a primary key but does not ensure that it is unique.

● Rejects events that have a null primary key value.

● Propagates insert, update, and delete opcodes as is, without modifying or validating them. Ensure that you validate these inserts, updates, and deletes elsewhere in the project.

● Does not detect duplicate inserts, bad updates, or bad deletes.

● Rejects events with upsert or safedelete opcodes.

● Treats all events as inserts for processing purposes, though they may contain different opcodes.

Supported Operations and Features

When a keyed stream is the target for the result of the operation, it supports:

● Inputs

● Computes

● Unions

● Pattern Matching

● Filters (see

Filters [page 27]

for details)

● Simple joins (see

Joins [page 28]

for details)

● Flex operations (see

Keyed streams support guaranteed delivery.

Unsupported Operations and Features

When the keyed stream is the target for the result of the operation, it does not support:

● Aggregations.

● Joins in which the only inputs are windows. You can get around this by performing a window-window join first, then feeding the results into a keyed stream.

Keyed streams reject:

● Upserts

● Safedeletes

● Any record with a primary key column that has a null value

For additional restrictions, see Inputs and Outputs [page 26]

, Joins [page 28] , and

Filters [page 27] .

Inputs and Outputs

Keyed streams can send to and receive from other streams (including other keyed streams), Flex operators, and windows. They can serve as inputs to relational operations like joins, aggregations, and computes, or as outputs to relational operations.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

Exceptions and considerations:

● A keyed stream cannot feed a window. Add a KEEP clause to the keyed stream to turn it into an unnamed window, which allows it to use memory and retain its state.

○ If you use a KEEP ALL clause, the unnamed window validates inserts, updates, and deletes.

○ If you use a KEEP clause with any other retention policy, the unnamed window treats updates as upserts and deletes as safedeletes. The unnamed window traps duplicate inserts unless the retention policy has allowed the original insert to be purged.

● When a stream feeds a keyed stream, the keyed stream produces inserts. When a keyed stream feeds a keyless stream, the stream follows its semantics of converting updates to inserts and silently dropping deletes.

● When a window feeds a keyed stream, the keyed stream outputs the inserts, updates, and deletes it receives with no changes.

● When a keyed stream feeds a Flex operator, the CCLScript code does not have access to the old record in the case of an update. The old record is always null.

● When a Flex operator feeds a keyed stream, the CCLScript code can generate only insert, update, and delete opcodes. Upsert and safedelete opcodes are not allowed.

● When a keyed stream feeds an event cache, the coalesce option is limited to the case when the records are coalesced on the key field.

Filters

When you use a WHERE clause, filter on columns that have values that do not change between an insert event and subsequent update and delete events. If the columns change, related events can be lost; for example, downstream elements might receive update or delete events without the insert that provided the data being updated or deleted, or fail to receive a delete for a previous insert.

In this example, we create two very similar elements, a keyed stream named KS1, and a window named W1:

CREATE OUTPUT STREAM KS1 PRIMARY KEY (Key1) AS SELECT In1.Key1,

In.Val1, In1.Val2 FROM In1 WHERE In1.Val1 > 10;

CREATE OUTPUT WINDOW W1 PRIMARY KEY (Key1) AS SELECT In1.Key1,

In.Val1, In1.Val2 FROM In1 WHERE In1.Val1 > 10;

Suppose In1 sends this data:

Keyed stream KS1 and window W1 produce different output:

Rather than filtering on Val1, which changes, filter on Val2, which does not. This approach provides more predictable results.

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

P U B L I C

Joins

When a keyed stream is the target of an inner join, ensure that the columns on which the join is performed do not change across an insert event and related update and delete events that follow. If the columns change, related events can be lost; the keyed stream may send update or delete events without the insert that provided the data being updated or deleted, or fail to send a delete for a previous insert.

Keyed streams are stateless except when performing a join with a window. In this type of join, the keyed stream uses memory to store a reference to the records in the window.

Restrictions on joins:

● A keyed stream can only be an outer member of an outer join (inner joins are supported).

● A keyed stream may not participate in a full join.

● When you join a keyed stream with a window, only events that arrive in the keyed stream trigger a join; changes to the window do not.

● A keyed stream cannot be the target of a join when all inputs to the join are windows (named or unnamed).

2.3.2 Windows

A window is a stateful element that can be named or unnamed, and retains rows based on a defined retention policy.

Create a window if you need data to retain state. To create a window, you can:

● Open the Streams and Windows compartment of the visual editor in SAP HANA studio, and click Input

Window.

● Use the CREATE WINDOW statement in the text editor. When creating the window, and to retain rows, assign a primary key

Since a window is a stateful element, with an underlying store, it can perform any operation specified by the opcode of an incoming event record. Depending on what changes are made to the contents of the store by the incoming event and its opcode, a window can produce output event records with different opcodes.

For example, if the window is performing aggregation logic, an incoming event record with an insert opcode can update the contents of the store and thus produce an event record with an update opcode. The same could happen in a window implementing a left join.

A window can produce an output event record with the same opcode as the input event record. For example, if a window implements a simple copy or a filter without any additional clauses, the input and output event records have the same opcode.

An incoming event record with an insert opcode can produce an output event record with a delete opcode. For example, a window with a count-based retention policy (for example, keeping five records) will delete those records from the store when the sixth event arrives, thus producing an output event record with a delete opcode.

Each subscribed window has a single thread that posts rows to all clients. If one subscriber to this window backs up and the client's queue is filled, it blocks subscription for all other clients.

In this section:

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

Retention Policies [page 29]

A retention policy specifies the maximum number of rows or the maximum period of time that data is retained in a window.

Named Windows [page 32]

A named window is explicitly created using a CREATE WINDOW statement, and can be referenced in other queries.

Unnamed Windows [page 33]

An unnamed window is an implicitly created stateful element that cannot be referenced or used elsewhere in a project.

Related Information

Streams [page 25]

2.3.2.1 Retention Policies

A retention policy specifies the maximum number of rows or the maximum period of time that data is retained in a window.

In CCL, you can specify a retention policy when defining a window. You can also create an unnamed window by specifying a retention policy on a window or keyed stream when it is used as a source to another element.

Retention is specified through the KEEP clause. You can limit the number of records in a window based on either the number, or age, of records in the window. These methods are referred to as count-based retention and time-based retention, respectively. You can also use the ALL modifier to explicitly specify that the window should retain all records.

Caution

If you do not specify a retention policy, the window retains all records. This can be dangerous: the window can keep growing until all memory is used and the system shuts down. The only time you should have a window without a KEEP clause is if you know that the window size will be limited by incoming delete events.

Including the EVERY modifier in the KEEP clause produces a jumping window, which deletes all of the retained rows when the time interval expires or a row arrives that would exceed the maximum number of rows.

Specifying the KEEP clause with no modifier produces a sliding window, which deletes individual rows once a maximum age is reached or the maximum number of rows are retained.

Note

You can specify retention on input windows (or windows where data is copied directly from its source) using either log file-based stores or memory-based stores. For other windows, you can only specify retention on windows with memory-based stores.

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

P U B L I C

Count-based Retention

In a count-based policy, a constant integer specifies the maximum number of rows retained in the window. You can use parameters in the count expression.

A count-based policy also defines an optional SLACK value, which can enhance performance by requiring less frequent cleaning of memory stores. A SLACK value accomplishes this by ensuring that there are no more than

<N>

<S>

rows in the window, where

<N>

is the retention size and

<S>

is the SLACK value. When the window reaches

<N>

<S>

rows, the system purges

<S>

rows. The larger the SLACK value, the better the performance, since there is less cleaning required.

Note

You cannot use the SLACK value in a jumping window's retention policy, because it cannot be used with the

EVERY modifier.

The default value for SLACK, 1, causes a significant impact on performance, because every new inserted record deletes the oldest record. Larger SLACK values improve performance by reducing the need to constantly delete rows.

Count-based retention policies can also support retention based on content/column values using the PER sub-clause. A PER sub-clause can contain an individual column or a comma-delimited list of columns. A column can only be used once in a PER sub-clause. Specifying the primary key or autogenerate columns as a column in the PER sub-clause results in a compiler warning, because these are unique entities for which multiple values cannot be retained.

This example creates a sliding window that retains the 100 most recent records that match the filter condition:

CREATE WINDOW Last100Trades PRIMARY KEY DEDUCED

KEEP 100 ROWS

AS SELECT * FROM Trades

WHERE Trades.Volume > 1000;

Once there are 100 records in the window, the arrival of a new record deletes of the oldest record in the window.

Adding a SLACK value of 10 means the window may contain as many as 110 records before any records are deleted:

CREATE WINDOW Last100Trades PRIMARY KEY DEDUCED

KEEP 100 ROWS SLACK 10

AS SELECT * FROM Trades

WHERE Trades.Volume > 1000;

This example creates a jumping window named TotalCost from the source stream Trades. This window retains a maximum of 10 rows, and deletes all 10 retained rows on the arrival of a new row:

CREATE WINDOW TotalCost

PRIMARY KEY DEDUCED

AS SELECT

trd.*,

trd.Price * trd.Size TotalCst

FROM Trades trd

KEEP EVERY 10 ROWS;

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

The next example creates a sliding window that retains two rows for each unique value of Symbol. Once two records have been stored for any unique Symbol value, the arrival of a third record (with the same value) deletes the oldest stored record with the same Symbol value:

CREATE SCHEMA TradesSchema (

Id integer,

TradeTime seconddate,

Venue string,

Symbol string,

Price float,

Shares integer );

CREATE INPUT WINDOW TradesWin1

SCHEMA TradesSchema

PRIMARY KEY(Id)

KEEP 2 ROWS PER(Symbol);

Time-based Retention

In a sliding window's time-based policy, a constant interval expression specifies the maximum age of the rows retained in the window. In a jumping window's time-based retention policy, all the rows produced in the specified time interval are deleted after the interval has expired.

This example creates a sliding window that retains each record received for 10 minutes. Each individual row that exceeds the retention time limit of 10 minutes is deleted:

CREATE WINDOW RecentPositions PRIMARY KEY DEDUCED

KEEP 10 MINS

AS SELECT * FROM Positions;

This example creates a jumping window named Win1 that keeps every row that arrives within a 100-second interval. When the time interval expires, all of the rows retained are deleted:

CREATE WINDOW Win1

PRIMARY KEY DEDUCED

AS SELECT * FROM Source1

KEEP EVERY 100 SECONDS;

The PER sub-clause supports content-based data retention, where data is retained for a specific time interval for each unique column value or combination. A PER sub-clause can contain a single column or a commadelimited list of columns, but you may only use each column once in the same PER clause.

Note

Time-based windows retain data for a specified time regardless of their grouping.

The next example creates a jumping window that retains five seconds worth of data for each unique value of

Symbol:

CREATE SCHEMA TradesSchema (

Id integer,

TradeTime seconddate,

Venue string,

Symbol string,

Price float,

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

P U B L I C

Shares integer );

CREATE INPUT WINDOW TradesWin2

SCHEMA TradesSchema

PRIMARY KEY(Id)

KEEP EVERY 5 SECONDS PER(Symbol);

Retention Semantics

When the insertion of one or more new rows into a window triggers the deletion of preexisting rows (due to retention), the window propagates the inserted and deleted rows downstream to relevant streams and subscribers. However, the inserted rows are placed before the deleted rows, since the inserts trigger the deletes.

Aging Policy

You can set an aging policy to flag records that have not been updated within a defined interval. This is useful for detecting records that may be stale. Aging policies are an advanced, optional feature for a window or other stateful element.

2.3.2.2 Named Windows

A named window is explicitly created using a CREATE WINDOW statement, and can be referenced in other queries.

Named windows can be derived or input:

● Derived windows can be:

○ Output, which can send data to an adapter.

○ Local windows, which are private and invisible externally. If a window does not have a qualifier, it is considered local.

● Input, which can send and receive data through adapters.

Both input and output windows are visible externally and can be subscribed to or queried.

The following table compares input, output, and local windows:

Type input output

Receives Data From

Input adapter or external appli cation that sends data into smart data streaming using the smart data streaming SDK.

Other windows, streams, or keyed streams.

Sends Data To

One or more of the follow ing: other windows, keyed streams, and output adapters.

Visible Externally

Yes

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

Type local

Receives Data From

Other windows, streams, or keyed streams.

Sends Data To

Other windows or keyed streams.

Visible Externally

2.3.2.3 Unnamed Windows

An unnamed window is an implicitly created stateful element that cannot be referenced or used elsewhere in a project.

Unnamed windows are implicitly created:

● When using a join with a window that produces a stream.

● When the KEEP clause is used with the FROM clause of a statement.

In both situations, when an unnamed window is created, it always includes a primary key.

Note

Although unnamed windows use additional memory, there is no memory reporting on them.

This example creates an unnamed window when using a join with a window:

CREATE INPUT WINDOW Win1 SCHEMA (Key1 INTEGER, Col1 STRING, Col2 STRING) PRIMARY

KEY (Key1);

CREATE INPUT WINDOW Win2 SCHEMA (Key1 STRING, Col3 STRING) PRIMARY KEY (Key1);

CREATE OUTPUT WINDOW Out1 PRIMARY KEY DEDUCED AS SELECT Win1.Key1, Win1.Col1,

Win1.Col2, Win.2.Col3

FROM Win1 INNER JOIN Win2 ON Win1.Col1 = Win2.Key1;

Note

The unnamed window is created to ensure that a join does not see records that have not yet arrived at the join. This can happen because the source to the join and the join itself are running in separate threads.

These four examples demonstrate when an unnamed window is created using a KEEP clause.

Example 1 creates an unnamed window on the input Trades for the MaxTradePrice window to keep track of a maximum trade price for all symbols seen within the last 10,000 trades:

CREATE WINDOW MaxTradePrice

PRIMARY KEY DEDUCED

STORE S1

AS SELECT

trd.Symbol, max(trd.Price) MaxPrice

FROM Trades trd KEEP 10000 ROWS

GROUP BY trd.Symbol;

Example 2 creates an unnamed window on Trades, and MaxTradePrice keeps track of the maximum trade price for all the symbols during the last 10 minutes of trades:

CREATE WINDOW MaxTradePrice

PRIMARY KEY DEDUCED

STORE S1

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

P U B L I C

AS SELECT

trd.Symbol, max(trd.Price) MaxPrice

FROM Trades trd KEEP 10 MINUTES

GROUP BY trd.Symbol;

Example 3 creates an unnamed window when using a window on a stream:

CREATE INPUT STREAM DataIn

SCHEMA (Symbol string, price money(2), size integer);

CREATE OUTPUT WINDOW MovingAvg

PRIMARY KEY DEDUCED AS SELECT DataIn.Symbol Symbol ,

avg(DataIn.price) AvgPrice ,

sum(DataIn.size) TotSize

FROM DataIn KEEP 5 MIN

GROUP BY DataIn.Symbol;

Example 4 creates a FiveMinuteVWAP unnamed window from the source stream Trades. Since the stream is an input to an aggregation, the unnamed window is created to allow the stream to have a retention policy:

CREATE INPUT STREAM Trades

SCHEMA (Tradeid integer, Symbol string, Price money(2), Shares integer)

CREATE WINDOW FiveMinuteVWAP

PRIMARY KEY DEDUCED AS SELECT

trd.Symbol, trd.Price, trd.Shares,

vwap(trd.Price, trd.Shares)

FROM Trades KEEP 5 MINUTES

GROUP BY trd.Symbol;

2.3.3 Input, Output, and Local

You can designate streams, windows, and keyed streams as input or derived. Derived streams and windows can be designated as either output or local.

Input and Output Streams and Windows

Input streams and windows can accept data from a source external to the project using an input adapter or by connecting to an external publisher. You can attach an output adapter or connect external subscribers directly to an input window or input stream. You can also use an ad-hoc query to select rows from an input window, insert rows in an input stream or insert, update, or delete rows in an input window.

Output windows, streams, and keyed streams can publish data to an output adapter or an external subscriber.

You can use an ad-hoc query to select rows from an output window.

Local streams, windows, and keyed streams are invisible outside the project and cannot have input or output adapters attached to them. You cannot subscribe to or use an ad-hoc query for the contents of local streams, windows, or keyed streams.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

Examples

This is an input stream with a filter:

CREATE SCHEMA mySchema (Col1 INTEGER, Col2 STRING);

CREATE INPUT STREAM IStr2 SCHEMA mySchema

WHERE IStr2.Col2='abcd';

This is an output stream:

CREATE OUTPUT STREAM OStr1

AS SELECT A.Col1 col1, A.Col2 col2

FROM IStr1 A;

This is an input window:

CREATE SCHEMA mySchema (Col1 INTEGER, Col2 STRING);

CREATE MEMORY STORE myStore;

CREATE INPUT WINDOW IWin1 SCHEMA mySchema

PRIMARY KEY(Col1)

STORE myStore;

This is an output window:

CREATE SCHEMA mySchema (Col1 INTEGER, Col2 STRING);

CREATE MEMORY STORE myStore;

CREATE OUTPUT WINDOW OWin1

PRIMARY KEY (Col1)

STORE myStore

AS SELECT A.Col1 col1, A.Col2 col2

FROM IWin1 A;

Local Streams and Windows

Use a local stream, window, or keyed stream when the stream does not need an adapter, or to allow outside connections. Local streams, windows, and keyed streams are visible only within the containing CCL project, which allows for more optimizations by the CCL compiler. Streams and windows that do not have a qualifier are local.

Note

A local window cannot be debugged because it is not visible to the SAP HANA Streaming Run-Test tools such as viewer or debugger.

Examples

This is a local stream:

CREATE SCHEMA mySchema (Col1 INTEGER, Col2 STRING);

CREATE LOCAL STREAM LStr1

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

P U B L I C

AS SELECT i.Col1 col1, i.Col2 col2

FROM IStr1 i;

This is a local window:

CREATE SCHEMA mySchema (Col1 INTEGER, Col2 STRING);

CREATE MEMORY STORE myStore;

CREATE LOCAL WINDOW LWin1

PRIMARY KEY (Col1)

STORE myStore

AS SELECT i.Col1 col1, i.Col2 col2

FROM IStr1 i;

2.3.4 Implicit Columns

All streams, windows, and keyed streams use three implicit columns called ROWID, ROWTIME, and

BIGROWTIME.

Column

ROWID

ROWTIME

BIGROWTIME

Datatype long seconddate bigdatetime

Description

Provides a unique row identification number for each row of incoming data.

Provides the last modification time as a date with second pre cision.

Provides the last modification time of the row with microsec ond precision. You can perform filters and selections based on these columns, like filtering out all of those data rows that occur outside of business hours.

You can refer to these implicit columns just like any explicit column (for example, using the stream.column convention).

2.3.5 Derived Streams and Windows

A derived stream or window is a the result of taking one or more streams or windows as input, and applying a continuous query.

A derived stream or window is the result of either a SELECT clause in a CREATE STREAM or CREATE WINDOW statement, or of a Flex operator applying CCLScript to events that arrive from one or more input streams or windows. Derived streams either direct data locally within a project, or publish the data to external destinations.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

2.4 Bindings on Streams, Keyed Streams, and Windows

Bindings enable data to flow between projects by allowing a stream, keyed stream, or window in one project to subscribe or publish to a stream, keyed stream, or window in another project.

A binding is a named connection from an input or output stream (or keyed stream or window) of one project to an input stream (or keyed stream or window) of another; you can configure it at either end.

● An input stream can subscribe to one or more streams in other projects. These streams do not have to be output streams—you can create an output binding on an input stream.

● An output stream can publish to one or more input streams in other projects. An output stream cannot receive incoming data, whether by subscription or publication.

Bindings reside in the CCR project configuration file, so you can change them at runtime. The streams being bound require compatible schemas.

Example

Binding to a Stream on an SSL-Enabled Cluster

This example shows a binding called BaseInputBinding that connects a local input stream called sin to a remote output stream that is also called sin. The <Manager> element that specifies the cluster hostname and port in the CCR file must include the https:// prefix, as shown here. If you omit the https:// prefix, the binding cannot pass data, so the input stream receives nothing.

<Password>PASSWORD</Password>

<Manager>https://CLUSTER_MANAGER_HOSTNAME:

CLUSTER_MANAGER_RPC_PORT</Manager>

</Managers>

</Cluster>

</Clusters>

<Cluster>cluster1</Cluster>

<BindingName>BaseInputBinding</BindingName>

</Binding>

</Bindings>

</Runtime>

</Configuration>

Example

Reconnection Intervals for Bindings

This example shows two bindings, b1 and b2, on a local input stream called MyInStream. The b1 binding includes a reconnection interval option specifying that if the connection between MyInStream and the remote output stream is lost, the project will attempt to reconnect every 10 seconds. Because the b2

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

P U B L I C

binding does not specify a reconnection interval, its reconnection attempts will occur at the default interval of five seconds:

<RemoteStream>MyInStream1</RemoteStream>

</Binding>

<RemoteStream>MyInStream2</RemoteStream>

</Binding>

</Bindings>

To suppress all reconnection attempts, set <ReconnectInterval> to 0. Use positive whole number values to set the reconnection interval in seconds.

Example

Configuring an Input Stream or Window to Provide Output

This example shows how to configure an input stream to send data to another input stream by setting the

<Output> parameter in the <Binding> element to true.

Note

Set the <Output> parameter to true only when you configure a binding on an input stream or window that is providing output. If you configure the binding on the stream or window that is receiving input, do not set the <Output> parameter. When you configure a binding on an output stream, you do not have to set the <Output> parameter, since output streams can only produce output.

In this example, output from the input stream MyInStream, in the local project, is bound to the input stream MyInStream1 in project p2. The line <Output>true</Output> tells the binding to publish (send data out) to the remote stream. Without that line, this binding would subscribe to data from MyInStream1 because bindings on input streams receive data by default:

<RemoteStream>MyInStream1</RemoteStream>

</Binding>

Example

Configuring a Window for Guaranteed Delivery

This example shows how to enable and configure guaranteed delivery (GD) on a window’s output binding.

The GD parameters are the same for input bindings.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

Enable GD for a binding to guarantee that if the connection between the binding and the remote stream is severed (by shutting down the project that contains the local stream, for example), all transactions that are supposed to be transmitted through the binding during its downtime are processed once the connection is re-established.

Use these parameters in the <Binding> element of your CCR file to set a binding to support guaranteed delivery:

● <EnableGD> – Specifies whether guaranteed delivery is enabled for this binding. Values are true and false.

Note

When you enable GD on a binding, make sure:

○ The binding’s source data window is running in GD mode or GD mode with checkpoint.

○ The binding’s target data window is backed by a log store.

● <GDName> – Supply a unique name for the GD session (subscription) this binding establishes.

● <GDBatchSize> – The number of transactions this binding may collect in a batch before releasing the batch to the target window. The binding issues a GD commit to the source data window after releasing the data. This setting is ignored when the source data window is in GD mode with checkpoint and the

<EnableGDCache> parameter on this binding is set to true. The default value is 10.

● <EnableGDCache> – Enable this binding to cache data. When the source data window is in GD mode with checkpoint, the binding receives checkpoint messages that indicate the last row of data that has been checkpointed by the window. If the binding is enabled for GD caching, it caches incoming transactions until it receives a checkpoint message from the source window. The checkpoint message triggers the binding to send all cached transactions up to the one indicated in the checkpoint message, to the target window. The binding issues a GD commit to the source data window after releasing cached data. If GD caching is disabled, checkpoint messages are ignored and the binding forwards data based on the value of <GDBatchSize>. The setting of <EnableGDCache> is ignored if the source data window is not in GD mode with checkpoint. Values are true (default) and false.

In this example, output from the local output stream MyOutStream is bound to MyInStream1 in project p1.

GD and GD caching are enabled. The GD session name is b1_GD1 and the GD batch size is 20 transactions:

<RemoteStream>MyInStream1</RemoteStream>

</Binding>

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

P U B L I C

2.5 Schemas

A schema defines the structure of data rows in a stream or window.

Every row in a stream or window has the same structure (schema), including the column names and datatypes and the order in which the columns appear. Multiple streams or windows can use the same schema, but each stream or window can only have one schema.

You can make a schema in one of the following ways:

● Create a named schema using the CREATE SCHEMA statement. Named schemas are useful when the same schema will be used in multiple places, since any number of streams and windows can reference a single named schema.

● Create an inline schema within a stream or window definition.

Example

Simple Schema CCL Example

This example uses a CREATE SCHEMA statement to create the named schema TradeSchema:

CREATE SCHEMA TradeSchema (

Ts BIGDATETIME,

Symbol STRING,

Price MONEY(4),

Volume INTEGER

);

This example uses a CREATE SCHEMA statement to make an inline schema:

CREATE STREAM trades SCHEMA (

Ts bigdatetime,

Symbol STRING,

Price MONEY(4),

Volume INTEGER

);

Related Information

Working with Schemas [page 97]

2.6 Adapters

Adapters connect SAP HANA smart data streaming to the external world.

An input adapter connects an input stream or window to a data source. It reads the data output by the source and modifies it for use in smart data streaming projects.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

An output adapter connects an output stream or window to a data sink. It reads the data output by the smart data streaming project and modifies it for use by the consuming application.

Adapters are attached to streams and windows using the ATTACH ADAPTER statement. Start them by using the ADAPTER START statement. Some projects may need to start adapters in a particular order, for example, to load reference data before attaching to a live event stream. Adapters can be assigned to groups and the

ADAPTER START statement can control the start-up sequence of the adapter groups.

See the SAP HANA Smart Data Streaming: Adapters Guide for detailed information about configuring individual adapters, datatype mapping, and schema discovery.

Related Information

Adding an Adapter to a Project [page 96]

Working with Stores [page 218]

SAP HANA Smart Data Streaming: Adapters Guide

2.7 Reference Table Queries

Reference table queries help augment the streaming data in a SAP HANA smart data streaming project with information from a table or view in SAP HANA.

This CCL element enables you to establish a reference to a table or view in SAP HANA from within the smart data streaming project, then use the reference in a join along with streams and windows. When an event arrives via a stream or window, the reference executes a query on the table in the external database and uses the returned data in the join to enrich streaming data with information from the database.

To create a reference, you need:

● The name of the database service to use.

● The name of the table from which to retrieve information.

● The schema of the table.

You can also specify:

● The primary key of the table as the primary key of the reference.

● That the reference should attempt to reconnect when the connection is lost.

● How many attempts to make.

● How long to wait between attempts.

In CCL, use the CREATE REFERENCE statement to define the reference, then use the FROM and ON clauses to join data from the reference with streams and windows in the smart data streaming project. You can also use the Reference shape (found under Streams and Windows in the palette) within the SAP HANA Streaming

Development perspective's visual editor.

In CCLScript, you can use an iterator over a reference from a local DECLARE block and in the Flex operator in the same way you use an iterator over a window. You may also iterate over a reference using the key search (if

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

P U B L I C

a primary key is defined), record-matching search, and for loop functionality. See the SAP HANA Smart Data

Streaming: CCL Reference for more information on these statements and clauses.

Related Information

SAP HANA Smart Data Streaming: CCL Reference

2.8 Stores

Set store defaults, or choose a log store or memory store to specify how data from a window is saved.

If you do not set a default store using the CREATE DEFAULT LOG STORE or CREATE DEFAULT MEMORY

STORE statements, each window is assigned to a default memory store. You can use the default store settings for store types and locations, if you do not want to assign new windows to specific store types.

Memory Stores

A memory store holds all data in memory. Memory stores retain the state of queries for a project from the most recent server start-up for as long as the project is running. Because query state is retained in memory rather than on disk, access to a memory store is faster than to a log store.

Use the CREATE MEMORY STORE statement to create memory stores. If no default store is defined, new windows are automatically assigned to a memory store.

Log Stores

The log store holds all data in memory, but also logs all data to the disk, guaranteeing data state recovery in the event of a failure. Use a log store to recover the state of a window after a restart.

Use the CREATE LOG STORE statement to create a log store. You can also set a log store as a default store using the CREATE DEFAULT LOG STORE statement, which overrides the default memory store.

Log store dependency loops can cause compilation errors when using log stores. You can create log store loops when you use multiple log stores in a project, and assign windows to these stores. For the best results, assign log stores to source windows only, or to assign all windows in a stream path to the same store. If you use logstore1 for n of those windows, then use logstore2 for a different window, do not reuse logstore1 further down the chain. For example, if Window Y assigned to Logstore B gets its data from Window X assigned to Logstore A, no window that (directly or indirectly) gets its data from Window Y should be assigned to

Logstore A.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

Related Information

2.9 Order of Elements

The order of CCL project elements, based on clause and statement syntax definitions and limitations, is very important to performance.

Define CCL elements that are referenced by other statements or clauses before using those statements and clauses; otherwise, you get compilation errors. You cannot reorder subclause elements within CCL statements or clauses.

The CCL compiler requires that the CCL objects (streams, windows, declare blocks, and so on) be defined before being referenced elsewhere. For example, if OutStream1 uses InStream1 as its input, the CCL compiler requires that InStream1 be defined before OutStream1. Otherwise, the compilation fails with an error stating that InStream1 does not exist.

Similarly, define a Global Declare Block containing CCLScript functions, variables, and parameters, before referencing them in any of the other CCL statements. The following is an example of incorrect order, which would cause the compiler to fail with an error stating that somefunc is not defined:

CREATE INPUT STREAM InStream1 SCHEMA (Col1 Integer, Col2 Integer);

CREATE OUTPUT STREAM OutStream1 AS SELECT InStream1.Col1, InStream1.Col2, somefunc(InStream1.Col1, InStream1.Col2) Result FROM InStream1;

DECLARE

integer somefunc(integer Col1, integer Col2) {

// …

}

END;

Instead, define a Declare Block for somefunc before referencing it in the projection for OutStream1.

Note

The only exception to this rule is that CCLScript functions within a Declare Block that reference each other may be defined in any order, and do not need to be defined before being referenced.

See the SAP HANA Smart Data Streaming: CCL Reference for more information.

Related Information

SAP HANA Smart Data Streaming: CCL Reference

SAP HANA Smart Data Streaming: Developer Guide

CCL Project Basics

P U B L I C

3 CCL Language Components

To ensure proper language use in your CCL projects, familiarize yourself with rules on case-sensitivity, supported datatypes, operators, and expressions used in CCL.

In this section:

Datatypes [page 45]

SAP HANA smart data streaming supports integer, float, string, money, long, and timestamp datatypes for all of its components.

Operators [page 49]

CCL supports various numeric, nonnumeric, and logical operator types.

Expressions [page 53]

An expression is a combination of one or more values, operators, and built-in functions that evaluate to a value.

CCL Continuous Queries [page 55]

Build a continuous query using clauses and operators to specify its function.

CCL Comments [page 57]

Like other programming languages, CCL lets you add comments to document your code. CCL recognizes two types of comments: doc-comments and regular multi-line comments.

Case-Sensitivity [page 57]

Some CCL syntax elements have case-sensitive names while others do not.

Literals [page 58]

The terms "literal" and "constant value" are synonymous and refer to a fixed data value. For example,

STOCK, SUDDEN ALERT, and 512 are all string literals; 1024 is a numeric literal.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

3.1 Datatypes

SAP HANA smart data streaming supports integer, float, string, money, long, and timestamp datatypes for all of its components.

Datatype bigdatetime

Description bigint binary

Timestamp with microsecond precision. The default format is YYYY-MM-

DDTHH:MM:SS:SSSSSS.

All numeric datatypes are implicitly cast to bigdatetime .

The rules for conversion vary for some datatypes:

● All boolean , integer , and long values are converted in their original format to bigdatetime .

● Only the whole-number portions of money(n) and float values are converted to bigdatetime . Use the cast function to convert money(n) and float values to bigdatetime with precision.

● All seconddate values are multiplied by 1000000 and converted to microsec onds to satisfy bigdatetime format.

● All msdate

values are multiplied by 1000 and converted to microseconds to sat isfy bigdatetime

format.

An alias for long .

boolean seconddate decimal

Represents a raw binary buffer. Maximum length of value is platform-dependent, with a size limit of 2 gigabytes. NULL characters are permitted.

Value is true or false. The format for values outside of the allowed range for boolean is 0/1/false/true/y/n/on/off/yes/no, which is case-insensitive.

Date with second precision. The default format is YYYY-MM-DDTHH:MM:SS.

double float

Used to represent numbers that contain decimal points. Accepts precision and scale , two mandatory parameters that determine the range of values that can be stored in a decimal field. precision specifies the total number (from 1 to 34) of digits that can be stored. scale specifies the number of digits (from 0 to precision ) that can be stored to the right of the decimal point.

The value 88.999p10s3 would have a decimal datatype of (10,3), which means the value has a decimal precision of 10 and a decimal scale of 3.

A 64-bit numeric floating point with double precision. The range of allowed values is approximately -10

308

through +10

308

. Equivalent to float .

A 64-bit numeric floating point with double precision. The range of allowed values is approximately -10

308

through +10

308

. Equivalent to double .

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

P U B L I C

Datatype integer interval long money

Description

A signed 32-bit integer. The range of allowed values is -2147483648 to +2147483647

(-2

to 2

-1). Constant values that fall outside of this range are automatically proc essed as long datatypes.

To initialize a variable, parameter, or column with a value of -2147483648, specify

(-2147483647) -1 to avoid CCL compiler errors.

A signed 64-bit integer that represents the number of microseconds between two timestamps. Specify an interval using multiple units in space-separated format, for example, "5 Days 3 hours 15 Minutes". External data sent to an interval column is assumed to be in microseconds. Unit specification is not supported for interval values converted to or from string

data.

When an interval

is specified, the value must fit in a 64-bit integer ( long

) when it is converted to the appropriate number of microseconds. For each interval

unit, the maximum allowed values that fit in a long

when converted to microseconds are:

● MICROSECONDS (MICROSECOND, MICROS): +/- 9223372036854775807

● MILLISECONDS (MILLISECOND, MILLIS): +/- 9223372036854775

● SECONDS (SECOND, SEC): +/- 9223372036854

● MINUTES (MINUTE, MIN): +/- 153722867280

● HOURS (HOUR,HR): +/- 2562047788

● DAYS (DAY): +/- 106751991

The values in parentheses are alternate names for an interval unit. When the maxi mum value for a unit is specified, specifying another unit causes an overflow. Specify each unit only once.

A signed 64-bit integer. The range of allowed values is -9223372036854775808 to

+9223372036854775807 (-2

to 2

63-1

To initialize a variable, parameter, or column with a value of -9223372036854775808, specify (-9223372036854775807) -1 to avoid CCL compiler errors.

A legacy datatype maintained for backward compatibility. It is a signed 64-bit integer that supports 4 digits after the decimal point. Currency symbols and commas are not supported in the input data stream.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

Datatype money(n) string time msdate

In this section:

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

Description

A signed 64-bit numerical value that supports varying scale, from 1 to 15 digits after the decimal point. The input data stream supports decimal points, but not currency symbols and commas.

The supported range of values change, depending on the specified scale: money(1) : -922337203685477580.8 to 922337203685477580.7

money(2) : -92233720368547758.08 to 92233720368547758.07

money(3) : -9223372036854775.808 to 9223372036854775.807

money(4) : -922337203685477.5808 to 922337203685477.5807

money(5) : -92233720368547.75808 to 92233720368547.75807

money(6) : -92233720368547.75808 to 92233720368547.75807

money(7) : -922337203685.4775808 to 922337203685.4775807

money(8) : -92233720368.54775808 to 92233720368.54775807

money(9) : -9223372036.854775808 to 9223372036.854775807

money(10) : -922337203.6854775808 to 922337203.6854775807

money(11) : -92233720.36854775808 to 92233720.36854775807

money(12) : -9223372.036854775808 to 9223,372.036854775807

money(13) : -922337.2036854775808 to 922337.2036854775807

money(14) : -92233.72036854775808 to 92233.72036854775807

money(15) : -9223.372036854775808 to 9223.372036854775807

To initialize a variable, parameter, or column with a value of

-92,233.72036854775807, specify (-9...7) -1 to avoid CCL compiler errors.

Specify explicit scale for money constants with Dn syntax, where n represents the scale. For example, 100.1234567D7, 100.12345D5.

Implicit conversion between money(n) types is not supported because there is a risk of losing range or scale. Perform the cast function to work with money types that have different scale.

Variable-length character string, with byte values encoded in UTF-8. Maximum string length is platform-dependent, with a size limit of 2 gigabytes. This size limit is reduced proportionally by the size of other content in the row, including the header.

Stores the time of day as a two-byte field with a range of 00:00:00 to 23:59:59. The default format is HH24:MM:SS.

A timestamp with millisecond precision. The default format is YYYY-MM-

DDTHH:MM:SS:SSS.

P U B L I C

Intervals [page 48]

Interval syntax supports day, hour, minute, second, millisecond, and microsecond values.

Decimal and Money Datatypes [page 49]

Although you can use either the decimal datatype or the money datatype to handle most financial data values, there are reasons for picking one datatype over the other.

3.1.1 Intervals

Interval syntax supports day, hour, minute, second, millisecond, and microsecond values.

Intervals measure the elapsed time between two msdate values, using 64 bits of precision. Results display in microseconds rather than milliseconds. You can create interval values in two ways:

● By declaring and populating a variable of type of interval

● By calculating the duration between two msdate values

All occurrences of intervals refer to this definition: value | {value [ {DAY[S] | {HOUR[S] | HR} | MIN[UTE[S]] | SEC[OND[S]] |

{MILLISECOND[S] | MILLIS} | {MICROSECOND[S] | MICROS} ] [...]}

If only value is specified, the msdate default is MICROSECOND[S]. Although you can specify multiple time units by separating each unit from the next with a space, you can specify each unit only once. For example, if you specify HOUR[S], MIN[UTE[S]], and SEC[OND[S]] values, you cannot specify these values again in the interval syntax.

Each unit has a maximum value when not combined with another unit:

Time Unit

MICROSECOND[S] | MICROS

MILLISECOND[S] | MILLIS

SEC[OND[S]]

MIN[UTE[S]]

HOUR[S] | HR

DAY[S]

Maximum Value Allowed

9,223,372,036,854,775,807

9,233,372,036,854,775

9,223,372,036,854

153,722,867,280,912

2,562,047,788,015

106,751,991,167

These maximum values decrease when you combine units.

Specifying value with a time unit means it must be a positive value. A negative value is treated as an expression. That is, -10 MINUTES in the interval syntax is treated as -(10 MINUTES). Similarly, 10

MINUTES-10 SECONDS is treated as (10 MINUTES)-(10 SECONDS).

For example:

3 DAYS 1 HOUR 54 MINUTES and

2 SECONDS 12 MILLISECONDS 1 MICROSECOND

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

You can only specify time units in CCL. When you specify values for the interval column using the API or an adapter you can only specify numeric values. These values are stored as a number in microseconds.

The following example calculates the difference, in microseconds, between two msdate values: msdate-value1

2015-01-01 14:12:00:002 msdate-value2

2015-01-01 14:12:001 msdate-value1 - msdate-value2

1000

3.1.2 Decimal and Money Datatypes

Although you can use either the decimal datatype or the money datatype to handle most financial data values, there are reasons for picking one datatype over the other.

The precision of the money datatype is set to 18, while the decimal datatype can have a precision up to 34, allowing it to handle a wider range of values. Use the decimal datatype if you know that your data will include very large values, or if you are uncertain about the data values you expect to receive or produce during processing.

Use the money datatype if you want better performance, and are confident that the data values you will be handling are within the money datatype's range.

Related Information

Datatypes in SQL Queries [page 300]

3.2 Operators

CCL supports various numeric, nonnumeric, and logical operator types.

Arithmetic Operators

Arithmetic operators are used to negate, add, subtract, multiply, or divide numeric values. They can be applied to numeric types, but they also support mixed numeric types. Arithmetic operators can have one or two arguments. A unary arithmetic operator returns the same datatype as its argument. A binary arithmetic operator chooses the argument with the highest numeric precedence, implicitly converts the remaining arguments to that datatype, and returns that type.

Operator

Meaning

Addition

Example Usage

3+4

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

P U B L I C

Operator

Meaning

Subtraction

Multiplication

Division

Modulus (Remainder)

Exponent

Change signs

Increment

Preincrement (++

) value is incremented before it is passed as an argument.

Postincrement (

++) value is passed and then incre mented.

Decrement

Predecrement (--

) value is decremented before it is passed as an argument.

Postdecrement (

--) value is passed and then decre mented.

Example Usage

7-3

3*4

8/2

8%3

4^3

-3

++a (preincrement) a++ (postincrement)

--a (predecrement) a-- (postdecrement)

Comparison Operators

Comparison operators compare one expression to another. The result of such a comparison can be TRUE,

FALSE, or NULL.

Comparison operators use this syntax: expression1 comparison_operator expression2

Operator

Meaning

Equality

Inequality

Greater than

Greater than or equal to

Less than

Less than or equal to

Member of a list of values. If the value is in the expression list's values, then the result is TRUE.

Example Usage a0=a1 a0!=a1 a0<>a1 a0!>a1 a0!>=a1 a0!<a1 a0!<=a1 a0 IN (a1, a2, a3)

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

Logical Operators

Operator

AND

NOT

XOR

Meaning

Returns TRUE if all expressions are TRUE, and FALSE otherwise.

Returns TRUE if all expressions are FALSE, and TRUE otherwise.

Returns TRUE if any of the expressions are TRUE, and FALSE otherwise.

Returns TRUE if one expression is TRUE and the other is FALSE. Returns

FALSE if both expressions are TRUE or both are FALSE.

Example Usage

(a < 10) AND (b > 12)

NOT (a = 5)

(b = 8) OR (b = 6)

(b = 8) XOR (a > 14)

String Operators

String operators are used to combine string expressions.

Operator

Meaning

Concatenates strings and returns another string.

Note

The + operator does not support mixed datatypes (such as an integer and a string).

Example Usage

'go' + 'cart'

LIKE Operator

The LIKE operator matches string expressions to strings that closely resemble each other, but do not match exactly. It can be used in column expressions and WHERE clause expressions.

Operator

Syntax and Meaning

Matches WHERE clause string expressions to strings that closely resem ble each other but do not match exactly:

Example Usage

Trades.StockName LIKE

"%Corp%" compare_expression LIKE pattern_match_expression

The LIKE operator returns a value of TRUE if compare_expression matches pattern_match_expression , or FALSE if it does not. The expressions can contain wildcards, where the percent sign (%) matches any length string, and the underscore (_) matches any single character.

[] Operator

The [] operator can be used with dictionaries and vectors, and to look up a window record by key field.

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

P U B L I C

Operator

[]

Syntax and Meaning

In the case of dictionaries, dictionary_name[key_value] references the value held in that dictionary location.

For vectors, vector_name[index] gets the value from a specific index position in a vector.

With an input window, you can use [] to look up a record in the window with a key field.

Example Usage rec := mydictionary[abc]; rec := myvector[2]

This example would get a record in Trades with the

TradeId key field having a value of 10:

Trades_stream[ [TradeId

= 10;|] ]

The next example would get a record called An other Event from another window which has all the key columns of the

Trades_stream, with the same key column name.

Trades_stream[AnotherE vent]

Order of Evaluation for Operators

When evaluating an expression with multiple operators, the engine evaluates operators with higher precedence before those with lower precedence. Those with equal precedence are evaluated from left to right within an expression. Use parentheses to override operator precedence, since the engine evaluates expressions inside parentheses before evaluating those outside.

Note

The ^ operator is right-associative. Thus, a ^ b ^ c = a ^ (b ^ c), not (a ^ b) ^ c.

The operators in order of preference are as follows. Operators on the same line have the same precedence:

● +.- (as unary operators)

● ^

● *, /, %

● +, - (as binary operators and for concatenation)

● =, !=, <>, <, >, <=, >= (comparison operators)

● LIKE, IN, IS NULL, IS NOT NULL

● NOT

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

● AND

● OR, XOR

3.3 Expressions

An expression is a combination of one or more values, operators, and built-in functions that evaluate to a value.

An expression often assumes the datatype of its components. You can use expressions in many places, including:

● Column expressions in a SELECT clause

● A condition of the WHERE clause or HAVING clause

Expressions can be simple or compound. A built-in function such as length() or pi() can also be considered an expression.

Simple Expressions

A simple CCL expression specifies a constant, NULL, or a column. A constant can be a number or a text string.

The literal NULL denotes a null value. NULL is never part of another expression, but NULL by itself is an expression.

To specify a column, include both the column name and the stream or window name, using the format source.column-name.

Some valid simple expressions include:

● stocks.volume

● 'this is a string'

● 26

Compound Expressions

A compound CCL expression is a combination of simple or compound expressions. Compound expressions can include operators and functions, as well as simple CCL expressions (constants, columns, or NULL).

Use parentheses to change the order of precedence of the expression's components.

Some valid compound expressions include:

● sqrt (9) + 1

● ('example' + 'test' + 'string')

● ( length ('example') *10 ) + pi()

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

P U B L I C

Sequences of Expressions

An expression can contain a sequence of expressions, separated by semicolons and grouped using parentheses, to be evaluated in order. The type and value of the expression is the type and value of the last expression in the sequence. The following example sets the values of the variables

<var1>

and

<var2>

, then returns the value 0.0:

(var1 := v.Price; var2 := v.Quantity; 0.0)

Datatypes in Expressions

Datatypes for column expressions are inherited from the schema, either as an explicitly created inline schema, or one discovered from the input adapter. You choose from supported datatypes in the schema editor, not in the column expression editor.

Enclose string data in expressions in single quotes, for example, 'my_string_data'.

Case Sensitivity

● All identifiers are case-sensitive. This includes names of streams, windows, parameters, variables, schemas, and columns.

● Keywords are case-insensitive, and cannot be used as identifier names.

● Built-in function names (except keywords) and user-defined functions are case-sensitive, however, some built-in function names have both lowercase and mixed-case forms, such as setOpcode and setopcode.

Conditional Expressions

A conditional CCL expression evaluates a set of conditions to determine its result. The outcome of a conditional expression is evaluated based on the conditions set. In CCL, the keyword CASE appears at the beginning of these expressions and follows a WHEN-THEN-ELSE construct.

The basic structure looks like this:

CASE

WHEN expression THEN expression

[...]

ELSE expression

END

The first WHEN expression is evaluated to be either zero or non-zero. Zero means the condition is false, and non-zero indicates that it is true. If the WHEN expression is true, the following THEN expression is carried out.

Conditional expressions are evaluated based on the order specified. If the first expression is false, then the subsequent WHEN expression is tested. If none of the WHEN expressions are true, the ELSE expression is carried out.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

A valid conditional expression in CCL is:

CASE

WHEN mark>100 THEN grade:=invalid

WHEN mark>49 THEN grade:=pass

ELSE grade:=fail

END

3.4 CCL Continuous Queries

Build a continuous query using clauses and operators to specify its function.

Syntax

select_clause from_clause

[matching_clause]

[where_clause]

[groupFilter_clause]

[groupBy_clause]

[groupOrder_clause]

[having_clause]

Clauses

All clauses in the following table are described in greater detail in the cited topics in the SAP HANA Smart Data

Streaming: CCL Reference.

Clause select_clause from_clause matching_clause where_clause groupFilter_clause groupBy_clause

Description

Defines the set of columns to be included in the output. See

SELECT Clause for more information.

Selects the source data is derived from. See FROM Clause for more information.

Used for pattern matching. See MATCHING Clause and Pat

tern Matching for more information.

Performs a filter. See WHERE Clause and Filters for more in formation.

Filters incoming data in aggregation. See GROUP FILTER

Clause and Aggregation for more information.

Specifies what collection of rows to use the aggregation op eration on. See GROUP BY Clause and Aggregation for more information.

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

P U B L I C

Clause groupOrder_clause having_clause

Description

Orders the data in a group before aggregation. See GROUP

ORDER BY Clause and Aggregation for more information.

Filters data that is output by the derived components in ag gregation. See HAVING Clause and Aggregation for more in formation.

Usage

CCL queries are embedded in the CREATE STREAM and CREATE WINDOW statements, and are applied to the inputs specified in the FROM clause of the query to define the contents of the new stream or window. The following example demonstrates the use of both the SELECT clause and the FROM clause as would be seen in any query.

Use the SELECT clause directly after the AS clause. The SELECT clause determines which columns or expressions the query has to use.

Following the SELECT clause, the FROM clause names the source used by the query. Following the FROM clause, implement available clauses to use filters, unions, joins, pattern matching, and aggregation on the queried data.

Example

This example obtains the total trades, volume, and VWAP per trading symbol in five-minute intervals:

SELECT

q.Symbol,

(trunc(q.TradeTime) + (((q.TradeTime - trunc(q.TradeTime))/300)*300))

FiveMinuteBucket,

sum(q.Shares * q.Price)/sum(q.Shares) Vwap,

count(*) TotalTrades,

sum(q.Shares) TotalVolume

FROM

QTrades q

Related Information

CCL Query Construction [page 143]

SAP HANA Smart Data Streaming: CCL Reference

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

3.5 CCL Comments

Like other programming languages, CCL lets you add comments to document your code. CCL recognizes two types of comments: doc-comments and regular multi-line comments.

The visual editor within the SAP HANA Streaming Development perspective recognizes a doc-comment and puts it in the comment field of the top-level CCL statement (such as CREATE SCHEMA or CREATE INPUT

WINDOW) immediately following it. Doc-comments that do not immediately precede a top-level statement are seen as errors by the visual editor.

Regular multi-line comments do not get treated specially by the studio and may be used anywhere in the CCL project.

Begin a doc-comment with /** and end it with */. For example:

/**

This is a doc-comment. Note that it begins with two * characters instead of one. All text within the begin and end tags is recognized by the Studio visual editor and associated with the immediately following statement (in this case the CREATE SCHEMA statement).

CREATE SCHEMA S1 ...

The CREATE SCHEMA statement provided here is incomplete; it is shown only to illustrate that the doccomment is associated with the immediately following CCL statement.

Begin a multi-line comment with /* and complete it with */. For example:

This is a multi-line comment.

All text within the begin and end tags is treated as a comment.

It is common to delineate a section of code using a row of asterisks. For example:

/*************************************************************

Do not modify anything beyond this point without authorization

**************************************************************/

CCL treats this rendering as a doc-comment because it begins with /**. To achieve the same effect using a multi-line comment, insert a space between the first two asterisks: /* *.

3.6 Case-Sensitivity

Some CCL syntax elements have case-sensitive names while others do not.

All identifiers are case-sensitive. This includes the names of streams, windows, parameters, variables, schemas, and columns. Keywords are case-insensitive, and cannot be used as identifier names. Adapter properties also include case-sensitivity restrictions.

Most built-in function names (except those that are keywords) and user-defined functions are case-sensitive.

While the following built-in function names are case-sensitive, you can express them in two ways:

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

P U B L I C

● setOpcode, setopcode

● getOpcode, getopcode

● setRange, setrange

● setSearch, setsearch

● copyRecord, copyrecord

● deleteIterator, deleteiterator

● getIterator, getiterator

● resetIterator, resetiterator

● businessDay, businessday

● weekendDay, weekendday

● expireCache, expirecache

● insertCache, insertcache

● keyCache, keycache

● getNext, getnext

● getParam, getparam

● seconddateInt, seconddateint

● intSeconddate, intseconddate

● uniqueId, uniqueid

● LeftJoin, leftjoin

● valueInserted, valueinserted

Example

Two variables, one defined as 'aVariable' and one as 'AVariable', can coexist in the same context, since they are treated as different variables. You can also define different streams or windows using the same name, but with different cases.

3.7 Literals

The terms "literal" and "constant value" are synonymous and refer to a fixed data value. For example, STOCK,

SUDDEN ALERT, and 512 are all string literals; 1024 is a numeric literal.

In smart data streaming, string literals are delimited by single (' ') quotation marks to distinguish them from object names, which are delimited by double (" ") quotation marks.

Neither BLOB nor XML datatypes have literals.

In this section:

Time Literals [page 59]

Use time literals to specify msdates and intervals.

Boolean Literals [page 60]

Boolean literals are true or false statements that are case-insensitive. For example, True, false, TRUE,

FALSE, tRuE, fAlSe, true, False, truE, and falsE are all valid.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

String Literals [page 60]

String literals appear as a part of expressions. String literals are also sometimes called character literals or text literals.

Numeric Literals [page 61]

Use numeric literals to specify integers, long, and floating-point numbers.

3.7.1 Time Literals

Use time literals to specify msdates and intervals.

MsDate Literals

The syntax of a msdate literal is:

<MSDATE> 'YYYY-MM-DD [HH:MI[:SS[.FF]]]'

Where:

● YYYY-MM-DD are numeric designations of the year, month, and day.

● HH:MI are numeric designations for hour and minute.

● :SS is a designation for seconds, used only if the hour and minute are specified.

● .FF is a designation for fractions of a second, using zero to six digits and only if seconds are specified.

Use one or more blank spaces to separate the date from the time specification.

Some valid msdates are:

MSDATE '2002-03-12 06:05:32.474003'

MSDATE '2005-02-01'

MSDATE '2003-11-30 15:04'

In some contexts, such as when putting row msdates into CSV files, you can enter msdates as a number of microseconds elapsed since midnight January 1, 1970. In this case, the numbers are treated as though they are relative to UTC, rather than local time. For example, if you use 1 as the msdate, and your local time zone is

Pacific Standard Time (eight hours behind UTC), the result is the following msdate:

1969-12-31 16:00:00.000001

Interval Literals

Use either of two formats for an interval literal. The first form is similar to that of msdate literals:

INTERVAL '{[D [day[s]]][ ][HH:MI[:SS[.FF]]]}'

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

P U B L I C

Where:

● D is the number of days. The space between the day specification and the hour and minute specification is optional.

● HH:MI are numeric designations for hour and minute.

● :SS is a designation for seconds, used only if the hour and minute are specified.

● .FF is a designation for fractions of a second, using zero to six digits and only if seconds are specified.

For example:

INTERVAL '999 days 23:59:59.999999'

The alternative syntax is:

{[D day[s]][ ][HH hour[s]][ ][MI minute[s]][ ][SS[.FF] second[s]][ ][NNN millisecond[s][ ][NNN microsecond[s>

All components of the interval are optional. For example:

4 minutes 5.6 seconds

Both forms of interval literals require that the values in each component be in the valid range. For example, you will get an error if you enter 61 minutes; you must enter this value as 1 hour 1 minute.

3.7.2 Boolean Literals

Boolean literals are true or false statements that are case-insensitive. For example, True, false, TRUE, FALSE, tRuE, fAlSe, true, False, truE, and falsE are all valid.

Values other than true or false ‒ such as Y/N or 1/0 ‒ are invalid.

3.7.3 String Literals

String literals appear as a part of expressions. String literals are also sometimes called character literals or text literals.

When a string literal appears as part of an expression in this documentation, it is indicated by the word TEXT.

The syntax for for both single-line and multi-line string literals is:

'character_string'

In all cases, character_string is a combination of alphabetic characters, numeric characters, punctuation marks, spaces, and tabs. In addition:

● Enclose both single-line and multi-line string literals in single (' ') quotation marks.

● Use double (" ") quotation marks as part of a string, if necessary.

● Use two adjacent single quotation marks with no character string between to represent an empty string.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

Note

Use double quotation marks to delimit a string for object names, not string literals. Object names cannot be multiple lines long, or contain newline (NL) characters or double quotation marks within the string.

To include a single quotation mark (or an apostrophe) in a string delimited by single quotation marks, enter a backslash before the quotation mark ( \' ) for the inside quotation marks. For example:

'And that\'s the truth.'

Double quotation marks do not require a delimiter. For example, the following string literal is valid:

'He said "No!"'

Some examples of valid string literals are:

'abc123'

'abc 123'

'It\'s a good idea.'

'"What?" he asked.'

Internationalization impacts string literals. All the literals in the preceding list are 7-bit ASCII literals, but this is also a literal:

'αβγ123'

3.7.4 Numeric Literals

Use numeric literals to specify integers, long, and floating-point numbers.

Integer Literals

Use the integer notation to specify integers in expressions, conditions, functions, and CCL statements.

The syntax of an integer literal is as follows, where integer refers to any whole numbers (including negatives) or zero:

[+|-]integer

Some valid integers are:

-45

+10023

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

P U B L I C

Long Literals

Long literals follow the same rules as integer literals. To force a literal that can be either integer or long into a long datatype, add the letter "L" to the end of the literal.

These are examples of valid long literals:

2147483648L

-2147483649L

-9223372036854775808L

Float Literals

A float literal is a floating-point number, usually used to represent numbers that include a decimal point. Use the float literal syntax whenever an expression requires a floating-point number.

The syntax of a float literal is as follows, where floating_point_number is a number that includes a decimal point:

[+|-]floating_point_number

[E[+|-]exponent]

The optional letter e or E indicates that the number is specified in scientific notation. The digits after the E specify the exponent, which can range from approximately -308 to +308.

Some valid float literals are:

1.234

-45.02

+10023.

.024

-7.2e+22

Note

Float values are accurate to 16 significant digits.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Language Components

4 Streaming Basics in SAP HANA Studio

The smart data streaming plugin for SAP HANA studio is an interactive development environment for building and testing event-based applications. It includes the SAP HANA Streaming Development and SAP HANA

Streaming Run-Test perspectives, as well as a visual and CCL editor.

The smart data streaming plugin for SAP HANA studio uses some Eclipse terms:

Term

View

Description

A named window in the studio user interface. Views are tab bed so several can overlap and occupy the same pane. The

SAP HANA Streaming Run-Test perspective, for example, includes the server view, manual input view, file upload view, and playback view. You can move, minimize, and resize views.

Perspective

A named set of views. Like views, perspectives are tabbed so several can use the same space. Perspectives in studio include SAP HANA Streaming Development and SAP HANA

Streaming Run-Test.

Workspace

The studio workspace is a directory that stores projects.

This workspace is distinct from cluster workspaces, which are logical groupings of deployed projects.

To access one of the smart data streaming perspectives, select Window Open Perspective Other , then open the desired perspective. The following table describes the activities you can perform in the different perspectives.

Perspective

SAP HANA Streaming Devel opment

Activities

● Create and edit projects.

● Develop projects and diagrams in the visual editor, a graphical editing environment.

● Develop projects in the CCL editor, a text-oriented editing environment where you edit

CCL code.

● Compile projects.

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

P U B L I C

Perspective

SAP HANA Streaming Run-

Test

Activities

● Start and connect to servers.

● Run projects.

● Enter test data by uploading data files to a server, or entering data manually to a stream.

● Publish data.

● Execute a query against a running project.

● Use the event tracer and debugger to set breakpoints and watchpoints, and trace the flow of data through a project.

● Record incoming event data to a playback file, and play back captured data into a run ning project.

● Monitor performance.

In this section:

Accessing Smart Data Streaming in SAP HANA Studio [page 64]

Access SAP HANA smart data streaming functionality within the SAP HANA studio by opening the SAP

HANA Streaming Run-Test and SAP HANA Streaming Development perspectives.

Streaming Development Perspective Components [page 65]

The visual editor, CCL editor, and other tools and views in the SAP HANA Streaming Development perspective allow you to create, view, and edit a diagram or CCL file.

Streaming Run-Test Perspective Components [page 78]

You can access various tools to test, monitor, debug, and fine-tune a project from the SAP HANA

Streaming Run-Test perspective.

Customizing the Streaming Work Environment [page 79]

Customize your smart data streaming plugin for SAP HANA studio interface to work the way you prefer.

The Studio Log File [page 82]

The SAP HANA studio logs activity and records it in a log file. Access this log file to view studio activity and to help troubleshoot events such as unexpected shut down.

4.1 Accessing Smart Data Streaming in SAP HANA Studio

Access SAP HANA smart data streaming functionality within the SAP HANA studio by opening the SAP HANA

Streaming Run-Test and SAP HANA Streaming Development perspectives.

Procedure

In SAP HANA studio, go to Window Open Perspective Other , then open the SAP HANA Streaming

Development or SAP HANA Streaming Run-Test perspective.

For more information on each perspective, see Streaming Basics in SAP HANA Studio [page 63] .

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

4.2 Streaming Development Perspective Components

The visual editor, CCL editor, and other tools and views in the SAP HANA Streaming Development perspective allow you to create, view, and edit a diagram or CCL file.

When viewing the SAP HANA Streaming Development perspective, its components are:

Editor

Palette

Overview

Includes groups of tools used to create new CCL elements on the diagram. Most shapes in the palette correspond to a CCL statement.

Provides a hierarchical tree structure of folders and files.

Project

Explorer

Properties view Displays the properties of the object selected in the diagram. You can also set properties in this view, and edit expressions.

Outline view Provides an index to all elements in the diagram as a hierarchical tree structure. Also shows the order in which adapters are started. Right-click an element in this view to show it in the diagram, delete it, modify it, or add a child element.

Helps you understand the big picture, and navigate easily to different areas of a large, complex diagram. For large diagrams you can scroll the editor by dragging the gray box in the overview.

Streaming Development perspective where you edit the diagram (in the visual editor) or

CCL (in the CCL editor). The visual and CCL text editors are completely integrated. When you save and switch to the other editor, your work is saved there as well.

Problems

Console

Provides full-text search capability for finding text strings in the workspace. Useful in navigating the file explorer, and project contents in the CCL editor. You can filter search results, and copy, remove, or replace results found.

Displays errors found when you compile a project.

Displays messages generated when interacting with smart data streaming components.

You can customize the arrangement of views in your perspectives. See

Customizing the Streaming Work

Environment [page 79]

In this section:

Visual Editor Authoring [page 66]

The visual editor lets you create and edit projects without learning CCL syntax. The editor is a valuable tool for experienced CCL programmers, particularly when working on complex projects, as a way to easily visualize the data flow and navigate within the project. In the visual editor, the project is represented by one or more diagrams that show streams, windows, adapters, and the data flows between them.

CCL Editor Authoring [page 73]

The CCL editor is a text authoring environment within SAP HANA Streaming Development perspective for editing CCL code.

Keyboard Shortcuts [page 77]

Use keyboard shortcuts to quickly access various functions within the visual or CCL editors.

Switching Between the CCL and Visual Editors [page 77]

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

P U B L I C

Change between the two editors to maximize flexibility for creating and editing a project.

Project Explorer [page 77]

Organize and navigate among your projects using the project explorer, which provides a treestructured hierarchy of folders and files.

4.2.1 Visual Editor Authoring

Begin by developing a simple project. Use the graphical tools to add streams and windows, connect them, and associate them with adapters. Add simple queries directly in the diagram using the visual editing tools. Once you have a basic diagram completed, compile and run your project.

When you are confident that your simple project is working, you can progress to advanced features for more complex queries, Flex operators for custom operations, modularity, and custom adapters. You can access many of these features in the visual authoring environment.

For more complex queries and other advanced features, you can switch to the CCL editor. A single CCL file can be open in only one editor at a time. The visual and CCL editors are completely integrated. When you save and switch to the other editor, your work is saved there as well.

In this section:

Diagrams [page 66]

In visual authoring, you use diagrams to create and manipulate the streams, windows, connections, and other components of a project, and create simple queries.

Shape Reference [page 68]

Each shape in the palette creates a specific type of stream or window, adapter, connection, reusable schema or module, or a store, to create a data flow.

Editing a Project in the Visual Editor [page 72]

In the visual editor, a diagram represents the elements of a project. Add or remove shapes and connections or edit shape attributes to make changes to a project.

4.2.1.1 Diagrams

In visual authoring, you use diagrams to create and manipulate the streams, windows, connections, and other components of a project, and create simple queries.

When you open a project in the visual editor, the project shows a collection of stream and window shapes that are connected with arrows showing the flow of data. You develop the project by selecting new input and output streams, windows, and other elements from the palette, dropping them onto the diagram, connecting them, and configuring their behavior.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

Every project has at least one diagram. A diagram in the visual editor is a projection of the associated CCL statements in the project.

When you add a shape or other element to a diagram, it is automatically added to the project when you save.

You can delete an element from a diagram only, or from the project.

Display diagrams in verbose or iconic mode: iconic compartments are collapsed to save space.

verbose all compartments in elements are visible.

● To expand or collapse all shapes in the diagram, use the All Verbose

or All Iconic buttons on the main toolbar.

○ To expand an individual shape, select it and click the "+" box in the shape toolbar.

○ To collapse an individual shape, select it and click the "-" box in the shape toolbar.

● To change the orientation, in the visual editor toolbar click Layout left to right

or Layout top down .

For more display options, right-click an object or the diagram surface and choose from the context menu.

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

P U B L I C

4.2.1.2 Shape Reference

Each shape in the palette creates a specific type of stream or window, adapter, connection, reusable schema or module, or a store, to create a data flow.

Shape

Connector

Note

Input Stream

Derived Stream (Local)

Derived Stream (Output)

Derived Window (Local)

Derived Window (Output)

Input Window

Flex

Purpose

Creates flows between streams and windows, establishes references be tween streams and shared compo nents, or attaches notes to shapes.

Usage

Click to select the connector tool, then click each of the shapes in the diagram to be connected..

Creates a comment on the diagram only. This comment does not appear in the CCL file.

The entry point for unkeyed event streams into a project. Receives data from either an input adapter or an ex ternal publisher.

Documents additional user generated information in the SAP HANA Stream ing Development perspective for a project in studio.

A stream does not retain any data and does not have a state. Data in an input stream is not keyed.

Applies a continuous query to data ar riving from another stream or window to produce a new stream.

Streams do not retain data and do not have keys. They are "insert only," meaning that their output consists only of inserts. Input must be a stream or a stream-window join.

By default, new streams (including de rived streams) are output, so they are visible to external subscribers. You can change the property to local.

Applies a continuous query to data ar riving from another stream or window.

Retains data, and retention rules can be set.

Data must be keyed so that every row has a unique key. Processes inserts, updates, and deletes both as local and output. You can use the toolbar to change the output to local, if you do not want it visible to external subscribers.

The entry point for event streams into a project where incoming events have primary keys and there is a desire to maintain a window of event data. Sup ports opcodes (insert, update, delete, upsert). Use this as an entry point for event streams if:

● The stream contains insert, up date and delete events, or,

● You need to retain a set of incom ing events.

Window size can be set by row count with a fixed number of input records, or by time with records being kept for a specified period. The window must be keyed, that is, every row must have a unique key value.

A programmable operator that uses custom CCLScript scripts to process incoming events.

A Flex operator can take input from any number of streams and/or windows and will produce a new derived stream or window (either local or output).

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

Shape

Aggregate

Compute

Filter

Join

Pattern

Union

Named Schema

Module

Log Store

Memory Store

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

Purpose

Takes input from a single stream or window and groups records using a common attribute. Produces a single output record for each group. Uses aggregate functions like sum(), count(), and so on.

Usage

Always creates a new window. Re quires a GROUP BY element. You can optionally set window size using reten tion rules.

Takes input from a single source and computes a new record for every re cord received. Allows you to change the schema on events, computing new fields and changing existing fields.

Produces a derived stream when the input is a stream. Produces a derived keyed stream when the input is a keyed stream. Produces a derived window when the input is a window.

Takes input from a single source and applies a filter. Creates a stream of re cords that match the filter criteria.

Produces a derived stream when the input is a stream. Produces a derived keyed stream when the input is a keyed stream. Produces a derived window when the input is a window.

Takes input from two or more sources and joins them based on common data elements.

See Joins [page 150] .

Takes input from two or more sources and detects patterns of events. One output record is produced every time a pattern is detected.

Detects patterns in the incoming data.

See Pattern Matching [page 159] .

Merges input from two or more sour ces. One output record is produced for every input record.

All inputs must have a common schema.

Represents a CCL CREATE SCHEMA statement. Reusable definition of col umn structure that can be referenced by streams and windows.

A schema defined outside of an object that can be used in multiple places, since any number of streams and win dows can reference a single named schema.

Represents a CCL CREATE MODULE statement. Creates a new module that can be used in one or more places in the project.

A module can contain all the same ele ments as a project and provides for re use.

Stores data held in windows. Provides disk-based recovery but is slower than a memory store.

By default, new windows are assigned to a memory store. Where recoverabil ity of data in a window is required, cre ate a log store and assign the window to it.

Stores data held in windows.

Faster than a log store but does not re cover data after shutdown:

● (Default) Created implicitly by the

CCL compiler, if no other store is specified.

● (Optional) Created explicitly, with windows assigned to specific stores, to optimize performance.

P U B L I C

Shape

Input Adapters

Output Adapters

Reference

Purpose

Connects an input stream or input window to an external data source.

Usage

Must be connected to either an input stream or input window. To use schema discovery—that is, to import the schema from the source—add the input adapter first, and then use schema discovery to create a con nected input stream or window with the imported schema.

Connects an output stream or window to a destination.

Must be connected to either an output stream or an output window.

A CCL element that establishes a ref erence from an external database ta ble to a project in smart data stream ing. Use reference—in joins and in

CCLScript programs—inside a module as well as within the main body of your project.

Can be used for data lookup and to en rich data streaming in SAP HANA smart data streaming with information permanently stored in an external da tabase table. For example, customer address and credit card information.

For schema discovery, datatypes in the source schema for the reference must be compatible with those found in the external database table the reference queries. If incompatible, unexpected behavior such as an abnormal shut down can occur.

In this section:

Adding Shapes to a Diagram [page 71]

Create streams, windows, and shared components, relate them using continuous queries, and attach them to adapters.

Adding Comments to Shapes [page 71]

Add comments to shapes in the visual editor that will appear within a tooltip when you hover over them.

Related Information

Connecting Elements [page 112]

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

4.2.1.2.1 Adding Shapes to a Diagram

Create streams, windows, and shared components, relate them using continuous queries, and attach them to adapters.

Procedure

1. Open a diagram in the visual editor.

2. Click a shape tool in the palette (Input Window, Flex, and so on), then click an empty area in the diagram to create the new shape.

3. To view actions needed to complete a shape definition, hover the mouse over the shape in the diagram.

The border changes from red to grey when the shape definition is complete and correct.

4.2.1.2.2 Adding Comments to Shapes

Add comments to shapes in the visual editor that will appear within a tooltip when you hover over them.

Prerequisites

Enable tooltip comments. Select the Show comments in shape tooltips checkbox in Window

Preferences SAP HANA smart data streaming CCL Editor .

Procedure

1. In the visual editor, select a shape you want to add a comment for by clicking on it.

2. After the shape is highlighted, select the comment field in the Properties view.

3. Click the ellipsis button and enter a comment into the box. Click OK.

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

P U B L I C

4.2.1.3 Editing a Project in the Visual Editor

In the visual editor, a diagram represents the elements of a project. Add or remove shapes and connections or edit shape attributes to make changes to a project.

Procedure

1. In the SAP HANA Streaming Development perspective, navigate to the project explorer.

2. Double-click the .cclnotation file name of your project to open it in the visual editor.

Note

If you see an error message indicating that the linked file does not exist, this is likely because the name of the .ccl linked to the .cclnotation file has changed. To open the .cclnotation file, right-click and choose Change Linked CCL File, then select the .ccl file to be used with this .cclnotation file.

3. Click in the diagram to begin editing using the palette.

Tip

To make the visual editor window full-screen, double-click the

<name>

:Diagram tab at the top. Doubleclick again to revert.

4. Save as you go (Ctrl+S).

This saves changes to both the .cclnotation file (the diagram) and the .ccl file (the CCL).

5. To toggle between the visual editor and the CCL editor, choose Switch to Text or Switch to Visual

(F4).

6. To close the diagram, press Ctrl+W or Ctrl+F4, or click the X on the tab at the top of the editor .

Note

The visual editor, like other graphical user interfaces, offers several ways to accomplish most tasks, although this guide may not list all of them. For example, in many contexts you can carry out an action by:

○ Clicking a button or other icon in a shape, or on the main toolbar.

○ Using a shortcut key.

○ Double-clicking an element to open it.

○ Right-clicking to select from the context menu.

○ Selecting from the main menu bar.

○ Editing element values in the Properties view.

SAP HANA studio also includes features common to Eclipse-based applications.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

4.2.2 CCL Editor Authoring

The CCL editor is a text authoring environment within SAP HANA Streaming Development perspective for editing CCL code.

You can work in the CCL editor exclusively, or use it as a supplement to the visual editor. The CCL editor offers syntax completion options, syntax checking, and error validation.

A single CCL file can be open in only one editor at a time. The visual and CCL editors are completely integrated: when you save and switch to the other editor, your work is saved there as well.

Most users new to smart data streaming find it easier to get started in the visual editor. As you gain experience with the product, and learn to successfully compile and run a simple project, you can use the CCL editor to add advanced features to your projects.

For example, you can add:

● Complex queries that exceed the capabilities of the visual editor.

● DECLARE blocks for declaring project variables, parameters, datatypes, and functions.

● CCLScript event handlers that you invoke with Flex operators.

● Reusable modules and schemas that can be used multiple times in a project, or across projects.

Several features simplify the process of editing CCL code in the CCL editor:

Feature

Completion Proposals

Case-Insensitive Syntax Highlighting

Error Validation/Syntax Checking

Description

Activate completion proposals in workspace [Ctrl + Space].

Done automatically when editing CCL code.

Access the Problems view to see errors in CCL code.

Compile and Report Compilation Errors Access the Problems view to see errors in CCL code.

For CCL language details, see the SAP HANA Smart Data Streaming: CCL Reference.

In this section:

Editing in the CCL Editor [page 74]

Update and edit CCL code as text.

Searching for Text [page 75]

Find text in CCL code.

Queries in CCL [page 75]

CCL queries are attached to derived streams or windows to select data from one or more inputs and transform it into the desired output.

CCL Functions [page 76]

A function is a self-contained, reusable block of code that performs a specific task.

Adding Tooltip Comments for the Visual Editor in CCL [page 76]

Write comments in CCL that appear as tooltips for shapes in the visual editor.

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

P U B L I C

Related Information

SAP HANA Smart Data Streaming: CCL Reference

4.2.2.1 Editing in the CCL Editor

Update and edit CCL code as text.

Procedure

1. Select the SAP HANA Streaming Development perspective.

2. In the project explorer, expand the project container, and double-click the .ccl file name to open it in the

CCL editor.

Note

Advanced CCL users can include multiple CCL files in the same project by using an IMPORT statement to import shared schemas and module definitions from another file. When using the IMPORT statement to import one CCL file into another, both CCL files must be contained within the same project. If not, you receive an error message stating that the file cannot be located.

3. Begin editing text in the CCL editor window.

Tip

If you open a .ccl file in the CCL editor when the same project is open in the visual editor, the CCL editor opens in read-only mode and you cannot edit the file.

Close both the visual editor and CCL editor for the project, and then reopen the project in the CCL editor.

Note

Backslashes within string literals are used as escape characters; specify any Windows directory paths with two backslashes.

4. (Optional) Press Ctrl+Space to show a syntax completion proposal.

5. (Optional) To insert CREATE statement template code, right-click, choose Create, and then choose the element to create.

6. Choose File Save (Ctrl+S) to save the .ccl file and the project.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

4.2.2.2 Searching for Text

Find text in CCL code.

Procedure

1. Choose Search File .

You can also start a new search from the link in the Search view, when no search results are visible.

2. Enter search criteria in the dialog.

3. Choose either:

○ Search to show results, or

○ Replace to replace results.

4. Review results in the Search view and choose from options in the Search toolbar.

Tip

Double-click a match to highlight it in the CCL editor.

4.2.2.3 Queries in CCL

CCL queries are attached to derived streams or windows to select data from one or more inputs and transform it into the desired output.

CCL embeds queries within CREATE STREAM and CREATE WINDOW statements in the same way that standard SQL uses CREATE VIEW statements. Unlike SQL, in CCL, SELECT is not a statement but rather is a clause used within a CREATE

<object-type>

statement.

Where the visual editor lets you select data using visual components referred to as simple queries, these queries are actually CCL statements that create a stream or window with an attached query.

To develop queries in CCL, refer to topics under

CCL Query Construction [page 143] , or see the Statements

and Clauses topics in the SAP HANA Smart Data Streaming: CCL Reference.

Related Information

SAP HANA Smart Data Streaming: CCL Reference

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

P U B L I C

4.2.2.4 CCL Functions

A function is a self-contained, reusable block of code that performs a specific task.

SAP HANA smart data streaming supports:

● Built-in functions, including aggregate, scalar, and other functions.

● User-defined CCLScript functions.

Built-in functions come with the software and include functions for common mathematical operations, aggregations, datatype conversions, and security.

Order of Evaluation of Operations

Operations in functions are evaluated from right to left. This is important when variables depend on another operation that must pass before a function can execute because it can cause unexpected results. For example: integer a := 1; integer b := 2; max( a + b, ++a );

The built-in function max(), which returns the maximum value of a comma-separated list of values, returns 4 since ++a is evaluated first, so max(4, 2) is executed instead of max(3, 2), which may have been expected.

For details, see the SAP HANA Smart Data Streaming: CCL Reference.

Related Information

SAP HANA Smart Data Streaming: CCL Reference

4.2.2.5 Adding Tooltip Comments for the Visual Editor in

CCL

Write comments in CCL that appear as tooltips for shapes in the visual editor.

If you want comments to appear as tooltips in the visual editor, insert a comment immediately preceding the declaration statement for the corresponding shape in this form:

/**InsertTooltipCommentHere*/

This is a CCL example of a tooltip comment for an input window shape in the visual editor:

/**InputWindowInStudio*/

CREATE INPUT WINDOW InputWindow1 ;

Comments added into the CCL editor in this manner appear as tooltips in the visual editor when you hover over the corresponding shapes.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

Note

Enable tooltip comments. Select the Show comments in shape tooltips checkbox in Window

Preferences SAP HANA smart data streaming CCL Editor .

4.2.3 Keyboard Shortcuts

Use keyboard shortcuts to quickly access various functions within the visual or CCL editors.

To show a list of all available keyboard shortcuts, press Ctrl+Shift+L.

To add a keyboard shortcut:

1. Select Window Preferences General Keys .

2. Select a command from the list.

3. In the Binding field, add the new shortcut.

4. Click OK.

4.2.4 Switching Between the CCL and Visual Editors

Change between the two editors to maximize flexibility for creating and editing a project.

Procedure

● To switch from the CCL editor to the visual editor, right-click and choose Switch to Visual (F6), or click in the main toolbar.

● To switch from the visual editor to the CCL editor, right-click in the diagram and choose Switch To Text

(F6), or click in the main toolbar.

4.2.5 Project Explorer

Organize and navigate among your projects using the project explorer, which provides a tree-structured hierarchy of folders and files.

The project explorer view lets you organize project files, navigate to files and perform various file-based actions:

● Create new CCL files

● Create new projects

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

P U B L I C

● Edit existing files

● Delete files

● Create new folders

The Toggle Streaming Nature option in the context menu toggles the appearance of smart data streamingspecific context menu options for individual projects in the project explorer. When using smart data streaming in the SAP HANA studio environment, use this option to enable or disable smart data streaming options as necessary.

Adding a new smart data streaming project displays it in the project explorer. As you add new elements to the project, such as new windows or streams, you can display them in the project explorer by selecting Refresh from the project explorer context menu.

4.3 Streaming Run-Test Perspective Components

You can access various tools to test, monitor, debug, and fine-tune a project from the SAP HANA Streaming

Run-Test perspective.

The components of the SAP HANA Streaming Run-Test perspective are:

Server View Start and connect to available servers. Your first project is there, already running.

Manual Input view Manually create and publish events as input to a stream or window.

Playback view Record data flowing into a running project, or play back recorded files.

File Upload view Publish an existing data file to an input stream or window.

SQL Query view Run a snapshot SQL query. It captures a snapshot of the current window state and displays results in the Console.

Console view Review log messages and other tracing and debugging information useful to developers.

Stream view Show the events of an output stream or the retained events in an output window of a running project.

Monitor view

Debugger view

Monitor performance of a running project.

Debug a project by setting breakpoints and watchpoints.

Event Tracer view Trace the flow of data through a project.

Other run-test tools include the Run Project button, which is also in the SAP HANA Streaming Development perspective.

In this section:

Project Execution and Testing [page 79]

Run and test all aspects of a streaming project using the SAP HANA Streaming Run-Test perspective.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

4.3.1 Project Execution and Testing

Run and test all aspects of a streaming project using the SAP HANA Streaming Run-Test perspective.

During development, you can use the SAP HANA Streaming Run-Test perspective to run any compiled project against a server, view data flowing through the streams and windows defined in the project, execute queries, and use debugging tools. Your project configuration and licensing determine the type of server connections you can use when running projects. Some adapters also have special licensing requirements.

In the studio you can connect immediately to a cluster to run projects, using default security established for smart data streaming during installation. A cluster consists of a group of server nodes, which are processes that run on hosts. A cluster can have a single node or multiple nodes.

In a production environment, you typically run projects on a server. Administrators monitor, manage, and configure smart data streaming server nodes, clusters, and projects using SAP HANA cockpit. Command-line utilities and procedures are also available, as discussed in the SAP HANA Smart Data Streaming: Configuration

and Administration Guide.

Related Information

Editing Preferences for Smart Data Streaming [page 80]

SAP HANA Smart Data Streaming: Configuration and Administration Guide

4.4 Customizing the Streaming Work Environment

Customize your smart data streaming plugin for SAP HANA studio interface to work the way you prefer.

Context

Note

As an Eclipse-based application, the smart data streaming plugin for SAP HANA studio automatically includes many features not specific to SAP HANA smart data streaming. Features documented here have been tested with the smart data streaming plugin for SAP HANA studio. Other Eclipse features may not work as expected. For example, the Team Synchronizing perspective is not supported.

In this section:

Edit preferences to customize the SAP HANA smart data streaming environment.

Manual Input Settings [page 81]

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

P U B L I C

Set default values on datatypes for data being published to a stream from the Manual Input view and the format in which the data is published.

Setting CCL Compiler Options [page 82]

You can adjust the CCL compiler options.

4.4.1 Editing Preferences for Smart Data Streaming

Edit preferences to customize the SAP HANA smart data streaming environment.

Context

You can also access many of these preferences from the related view in the smart data streaming plugin for

SAP HANA studio.

Procedure

1. Choose Window Preferences .

2. Expand SAP HANA smart data streaming, then expand to the preferences you want to set. All preference settings are optional.

CCL Editor Settings

Run Test

Compiler Options

Set syntax coloring and template options.

Set defaults for server connections, add new connections, set limits and filters for the stream view and server view, and set other options for running projects in studio, including the default HANA service entry.

Change the directory for the CCL compiler output (default is bin folder in your

<workspace\project>

folder).

Data Input Settings

Manual Input Settings

Set file upload and SQL Query view options.

Choose settings for the publishing data from manual input view, including defaults for all datatypes except money types.

Set defaults for the money(n) datatype.

Manual Input Settings -

Money Types

Network Connections

Shapes General

Specify how Studio will connect to other machines on the network.

Choose defaults for creating and displaying shapes in diagrams.

3. On each preference dialog, either:

○ Click Apply to save the new settings, or,

○ Click Restore Defaults to revert any changes you make.

Only the settings in the current dialog are applied or restored.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

4. Click OK to exit the Preferences dialog.

4.4.2 Manual Input Settings

Set default values on datatypes for data being published to a stream from the Manual Input view and the format in which the data is published.

Settings for most datatypes are in Manual Input Settings preferences. Settings for the money(n) datatype are in Manual Input Settings - Money Types preferences.

Note

When you double-click an input field containing a negative value, only the text up to the minus symbol is selected. If you want to select the value including the minus symbol, single-click and highlight the text. This affects the Manual Input Settings and Manual Input Settings - Money Types preferences, and the Manual

Input view in studio.

Setting

Publish Multiple Rows

Use Current Date

Description

Indicates whether data from an input stream is published in single instances or as multiple rows.

Indicates whether data should be published under the current date or maintain its historical date.

Interpret Date values in Manual Input and Stream Viewer as UTC

Indicates whether Manual Input date values are interpreted as UTC or in the local time zone.

Note

This has no effect on the Playback tool.

binary boolean string integer float long interval seconddate bigdatetime msdate

Indicates a binary value to be published to a stream. Use this setting to monitor the binary value of a stream by placing a traceable value in the field.

May be set to True or False.

Indicates the default value studio accepts for string types.

Indicates the default value studio accepts for integer types. Does not accept val ues with decimal points.

Indicates the default value studio accepts for float types.

Indicates the default value studio accepts for long types.

Indicates the default value studio accepts for interval types.

Indicates the default value for seconddate types. Click Select to open a calendar dialog and choose a default date with second precision.

Indicates the default value for bigdatetime types. Click Select to open a calendar dialog and choose a default bigdatetime with microsecond precision.

Indicates the default value for msdates types. Click Select to open a calendar di alog and choose a default timestamp with millisecond precision.

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

P U B L I C

Setting time

Description

Indicates the default value studio accepts for time types.

Note

You see an error message at the top of the preference window when you enter incorrect characters, or exceed the number of allowed characters in the field.

Related Information

Manually Entering Data to a Stream [page 125]

4.4.3 Setting CCL Compiler Options

You can adjust the CCL compiler options.

Procedure

1. Choose Window Preferences .

2. Click SAP HANA smart data streaming Run Test , and go to Compiler output directory.

3. To change the directory for your compiled projects, click Change..., select a directory, and click OK.

4. To confirm any other changes, click OK.

Note

By default, the compile directory is set to bin, which means the .ccx files are created in a subdirectory relative to the project's directory.

4.5 The Studio Log File

The SAP HANA studio logs activity and records it in a log file. Access this log file to view studio activity and to help troubleshoot events such as unexpected shut down.

The studio log file resides in your workspace directory under workspace/.metadata/.log. View the log within studio:

1. Select Help > About Studio.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

2. Click Configuration Details.

3. Click Configuration.

4. Click View Error Log.

5. If prompted, select a text editor to view the file with.

The log provides read-only details on internal studio activity. You cannot modify the file to change what it reports on, or its level of verbosity.

SAP HANA Smart Data Streaming: Developer Guide

Streaming Basics in SAP HANA Studio

P U B L I C

5 Cluster Connectivity in Studio

Before you can run a project in SAP HANA studio, first ensure that you are connected to an SAP HANA smart data streaming server.

A cluster consists of one or more workspaces, each with one or more projects. These projects can be running or stopped. All workspaces are within one server, which allows you to simultaneously work with multiple projects. You can also share a project within the cluster with other users. You can adjust the settings of the clustered projects through studio. However, you cannot use studio to change streaming node (host) settings.

When you click the run button, by default, studio will perform your most recent run operation. Alternatively, you can right-click to specify a different option.

In this section:

Changing Networking Preferences [page 84]

Modify the default preferences for how the machine running SAP HANA studio connects with other smart data streaming machines.

Configuring a Cluster Connection [page 85]

Set preferences to manage and run cluster connections with the smart data streaming plugin for SAP

HANA studio.

Connecting to a Cluster [page 87]

Connect to an SAP HANA smart data streaming server from SAP HANA studio to be able to run a project.

Modifying a Cluster Connection [page 87]

Change the authentication settings of an existing, configured cluster connection.

Managing Secure Storage Credentials for a Streaming Server Connection [page 88]

Use secure storage to encrypt username and password credentials for an SAP HANA smart data streaming server connection in studio. Secure storage enables credential information for the streaming server connections to persist in studio.

5.1 Changing Networking Preferences

Modify the default preferences for how the machine running SAP HANA studio connects with other smart data streaming machines.

Context

SAP HANA studio sets the Active Provider to Direct to guarantee that network connections do not use a proxy server. If your network requires a different setting (such as the Eclipse default of setting it to Native if present, or Manual otherwise), modify the network preferences for studio.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Cluster Connectivity in Studio

Note

If the network is configured to use proxy servers, set the no_proxy environment variable on all publisher and subscriber machines in order to ensure successful publishing and subscribing of data.

Procedure

1. Open the studio.

2. Select Preferences General Network Connections .

3. Set the connection options as required for your network. If unsure, confirm the settings with your system or network administrator. To connect to a Web service through a proxy server:

○ Select Native from the

Active Provider

dropdown menu to use the system configured proxy settings.

○ Select Manual to configure your own proxy settings:

○ Set the proxy information for the HTTP or HTTPS schema.

○ Set up proxy bypass for certain hosts. Add both the fully qualified domain name and short name of each host to the

Proxy bypass

list.

If you set

Active Provider

to Native or Manual, and you are connecting to the server using a native connection (directly rather than through a Web service), ensure both the fully qualified domain name and the short name of the server host are added to the

Proxy bypass

list. Otherwise, the connection may fail.

4. Click Apply to save your new settings, then click OK.

5.2 Configuring a Cluster Connection

Set preferences to manage and run cluster connections with the smart data streaming plugin for SAP HANA studio.

Prerequisites

The administrator of the cluster has provided the cluster host name and port number.

Procedure

1. To add a new cluster connection, open the SAP HANA Streaming Run-Test perspective, and select New

Server URL in the server view toolbar.

SAP HANA Smart Data Streaming: Developer Guide

Cluster Connectivity in Studio

P U B L I C

Note

Alternatively, in the server view toolbar, you can also select Studio Preferences and add a new connection through SAP HANA smart data streaming Run Test , and click Run Test. Select

New.

2. In the New Server dialog, enter the host name and port number for the cluster connection.

Specify the port number using the format 3xx26, where xx is the instance number of your SAP HANA system. For example, if the instance number of your SAP HANA system is 00, enter the port number for the streaming server connection as 30026. Similarly, if your instance number is 14, then the port number becomes 31426.

3. Select a connection protocol: Native, or Web Services.

○ If you select Web Services, edit your network connection preferences. See

Changing Networking

Preferences [page 84] to configure a proxy server.

○ You can only connect through Web Services using the REST port. Before selecting this option, configure the Web Services Provider in the wsp.xml file. See Configuring the Web Services Provider in the SAP HANA Smart Data Streaming: Adapters Guide.

○ Connecting through a Web service is primarily intended for scenarios where a client is behind a firewall. Otherwise, use a native connection when possible, for performance reasons.

4. To enable encryption for Cluster Manager connections, select SSL.

5. Select User/Password as the authentication method. Kerberos and RSA are not supported for streaming clusters.

6. Click OK.

Results

In the SAP HANA Streaming Run-Test perspective, the server view accesses the list of stored server connections, and studio shows a login dialog for your clusters.

Note

To connect all listed servers, select Reconnect All in the server View toolbar.

Related Information

SAP HANA Smart Data Streaming: Adapters Guide

SAP HANA Smart Data Streaming: Configuration and Administration Guide

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Cluster Connectivity in Studio

5.3 Connecting to a Cluster

Connect to an SAP HANA smart data streaming server from SAP HANA studio to be able to run a project.

Prerequisites

Start your SAP HANA system, and ensure the SAP HANA smart data streaming server on that system is running.

Procedure

1. Select the SAP HANA Streaming Run-Test perspective.

The server view opens, displaying a list of the available clusters.

2. Right-click on the entry for the cluster you want (for example, myserver.mycompany.com:12345).

Studio displays a pop-up menu.

3. Select Connect Server. If prompted, supply your SAP HANA username and password.

The server view displays the workspaces on the cluster and the projects in each workspace.

4. Right-click on the project you want to run.

Studio displays a pop-up menu.

5. Select Show in from the menu.

Studio displays a pop-up menu listing ways to view the project's progress.

6. Select the viewing method, for example Event Tracer View.

Studio starts displaying the project's progress in the specified view.

5.4 Modifying a Cluster Connection

Change the authentication settings of an existing, configured cluster connection.

Context

If the administrator of the cluster changes its authentication settings, modify the cluster connection in studio accordingly.

SAP HANA Smart Data Streaming: Developer Guide

Cluster Connectivity in Studio

P U B L I C

Procedure

1. In the server view of the SAP HANA Streaming Run-Test perspective, select Studio Preferences SAP

HANA smart data streaming Studio Run Test .

Studio displays the Run Test screen.

2. Select an existing server connection.

The Remove and Edit buttons are activated.

3. Click Edit.

Studio displays the Remote Server Connection screen.

4. Make your changes and click OK.

Studio displays the Run Test screen.

5. Click OK to save your changes.

5.5 Managing Secure Storage Credentials for a Streaming

Server Connection

In this section:

Storing Credentials in Secure Storage for a Streaming Server Connection [page 89]

Store username and password credentials in secure storage for smart data streaming server connections. Secure storage stores credentials so that you do not have to enter username and password information each time you connect to a server. Studio stores the connections securely so your credentials are safe.

Modifying Credentials in Secure Storage for a Streaming Server Connection [page 90]

Modify username and password credentials in secure storage for an SAP HANA smart data streaming server connection. Change username and password information as required.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Cluster Connectivity in Studio

5.5.1 Storing Credentials in Secure Storage for a Streaming

Server Connection

Store username and password credentials in secure storage for smart data streaming server connections.

Secure storage stores credentials so that you do not have to enter username and password information each time you connect to a server. Studio stores the connections securely so your credentials are safe.

Context

Studio prompts you to enter username and password credentials in the following situations:

● The first time you connect to a server in studio.

● If you choose not to persist credential information.

● If the credential information in secure storage has been reset or removed.

When you connect to a server in studio, the Set Credentials prompt dialog contains the checkbox Store

Smart Data Streaming Credentials in Secure Storage. Click the checkbox to store the username and password information you entered. If you do not choose to store the credentials using the Store Smart Data

Streaming Credentials in Secure Storage option, you can store the credentials by opening the Preferences dialog.

Store credential information in secure storage.

Procedure

1. In the main menu, select Windows Preferences .

2. Select SAP HANA smart data streaming Run-Test .

3. Click on the checkbox for Use Secure Storage for Streaming Credentials.

4. Click Apply then OK.

The credentials you entered when prompted are saved in C:\Users\

<username> \.eclipse

\org.eclipse.equinox.security\secure_storage.

SAP HANA Smart Data Streaming: Developer Guide

Cluster Connectivity in Studio

P U B L I C

5.5.2 Modifying Credentials in Secure Storage for a

Streaming Server Connection

Modify username and password credentials in secure storage for an SAP HANA smart data streaming server connection. Change username and password information as required.

Procedure

1. In the main menu, select Windows Preferences .

2. Open the Secure Storage preference page. Either:

○ Select Smart Data Streaming Run-Test and click on the Secore Storage Settings link, or,

○ Select General Secutiry Secure Storage .

3. Click on UI Prompt; then click on Change Password.

4. Modify the password information.

Passwords are case-sensitive. You can use any alpha-numeric combination as well as symbols.

5. Click Yes or No for Password Recovery.

Password Recovery prompts you to provide two questions and their expected answers. Once enabled, you can click on Password Recovery, answer the two questions, and recover your password, which avoids having to change the password. You can only access Password Recovery from the Secure Storage preference page.

6. Click Finish.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Cluster Connectivity in Studio

6 Developing Streaming Projects

Use either the visual editor or the CCL editor within the SAP HANA Streaming Development perspective to create and modify your projects. Start by developing a simple project, then test it iteratively as you gradually add greater complexity.

Context

You can create and edit a streaming project in the SAP HANA studio using either the visual editor or the CCL editor. When you create a new project, smart data streaming creates a .ccl file and a .cclnotation file automatically. You can edit either file in its respective editor; all changes are reflected in the parallel file. For example, if you add and configure an input adapter in the visual editor, then switch to the CCL editor, you see the ATTACH INPUT ADAPTER statement with all configured properties.

Procedure

1. Create a new streaming project. See Creating a Streaming Project in SAP HANA Studio [page 93]

2. Add input streams and windows. See Adding an Input Stream or Window to a Project [page 108]

3. Add output streams and windows with simple continuous queries. Gradually add complexity. See

Simple

Queries [page 144] .

4. Attach adapters to streams and windows to subscribe to external sources or publish output. See Adding an Adapter to a Project [page 96]

5. (Optional) Use functions in continuous queries to perform mathematical operations, aggregations, datatype conversions, and other common tasks:

○ Built-in functions for many common operations

○ User-defined functions written in the CCLScript programming language

6. (Optional) Create named schemas to define a reusable data structure for streams or windows. See

Working with Schemas [page 97] .

7. (Optional) Create memory stores or log stores to retain the state of data windows in memory or on disk.

See Working with Stores [page 218]

8. (Optional) Create modules to contain reusable CCL that can be loaded multiple times in a project. See

Modularity [page 185]

9. Compile the project. See

Compiling a Streaming Project [page 116]

10. Run the compiled project against test data, using the debugging tools in the smart data streaming in SAP

HANA studio and command line utilities. See

Running and Testing Streaming Projects [page 119] .

Repeat this step as often as needed.

In this section:

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

Creating a Streaming Project in SAP HANA Studio [page 93]

Create and save streaming projects in the SAP HANA studio workspace directory.

Renaming a Project [page 95]

Change the streaming project name for easier recognition, or to reflect any changes in the content.

Deleting a Project [page 96]

Delete unused streaming projects from the smart data streaming plugin for SAP HANA studio.

Adding an Adapter to a Project [page 96]

Attach an adapter to a project by inserting it into the project diagram, connecting it to a stream or window, and setting properties.

Working with Schemas [page 97]

Discover external schemas and create CCL schemas, streams, or windows based on the format of the data from the data source that is connected to an adapter.

Adding an Input Stream or Window to a Project [page 108]

You can add input streams and windows to a project, which accept data from a source external to the streaming project.

Connecting Elements [page 112]

The keep policy determines the basis for retaining rows in a window.

Connect two shapes in a diagram to create a data flow or link between them.

Setting Key Columns [page 113]

Set primary keys in the visual editor within the column compartment of the window, and Flex operator shapes.

Editing Column Expressions [page 114]

Modify column expressions for windows and streams using an inline editor or dialog-based expression editor.

Deleting an Element [page 114]

Delete an element to remove it from the project completely, or delete it from the diagram while retaining it in the project.

Creating Windows and Streams from a Database Table [page 115]

Create windows and streams from SAP HANA tables for a smart data streaming project in studio.

Using the SAP HANA Repository [page 115]

There are various advantages to placing your smart data streaming projects into the SAP HANA repository, including version control, and the ability to share your project files with other team members and create delivery units. Create a new smart data streaming project and share it to create a link between your development workspace and the SAP HANA repository.

Compiling a Streaming Project [page 116]

When you compile a project, smart data streaming produces an executable .ccx file from CCL code.

This file is necessary to run a project on smart data streaming.

Compiling Projects When Running SAP HANA Studio on Mac OS [page 117]

Before you can compile and run a streaming project in SAP HANA studio on Mac OS, or on Linux with no smart data streaming server installed, first connect to an SAP HANA smart data streaming server.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

6.1 Creating a Streaming Project in SAP HANA Studio

Create and save streaming projects in the SAP HANA studio workspace directory.

Procedure

1. In the SAP HANA studio menu bar, go to File New Project... , then select Smart Data

Streaming New Streaming Project ,

2. In the Name field, enter a valid project name. Create a name that:

○ Starts with a lowercase letter, underscore, or dollar sign.

○ Uses lowercase letters, numbers, underscores, or dollar signs for all other characters.

○ Cannot contain spaces.

3. In the Directory field, accept the default location or browse to a directory in which to store the new project folder.

4. Click Finish to create the project files.

The new project opens in the visual editor with one input stream, NEWSTREAM, and an inline schema ready for editing.

In this section:

Opening a Streaming Project in SAP HANA Studio [page 94]

Open an existing streaming project in the SAP HANA studio.

Importing a Streaming Project into SAP HANA Studio [page 94]

If you have streaming projects that are not located in your current workspace, import them into the

SAP HANA workspace.

SAP HANA Navigator [page 95]

Use the SAP HANA Navigator to select tables and views within SAP HANA, then drop them directly into a project diagram in the studio visual editor. This creates a new adapter reference or named schema in the streaming project.

Related Information

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

6.1.1 Opening a Streaming Project in SAP HANA Studio

Open an existing streaming project in the SAP HANA studio.

Procedure

1. In the SAP HANA studio menu bar, click Windows Open Perspective Other SAP HANA

Streaming Development .

2. Click OK.

3. In the studio toolbar, click the Open Streaming Project icon.

4. Browse to the root directory of the project.

5. (Optional) Select Copy projects into workspace to copy and open the project in the workspace. Changes are made to the copy only.

If you do not select this option, the project opens in its original location.

6. Click Finish.

6.1.2 Importing a Streaming Project into SAP HANA Studio

If you have streaming projects that are not located in your current workspace, import them into the SAP HANA workspace.

Procedure

1. In the SAP HANA studio menu bar, click Windows Open Perspective Other .

2. From the Open Perspective dialog, select SAP HANA Streaming Development and click OK.

3. In the SAP HANA Streaming Development perspective, right-click the project explorer view, and select

Import from the context menu.

4. In the Import dialog, expand the General folder and click Existing Projects into Workspace, then click

Next.

5. Enable the Select root directory option and enter or browse to the root directory containing the projects you are importing.

6. (Optional) Clear the check mark from any projects you do not want to import.

7. (Optional) Clear the Copy projects into workspace option.

8. Click Finish.

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

6.1.3 SAP HANA Navigator

You can then use the new reference in a join along with SAP HANA smart data streaming streams and windows. When an event arrives via a stream or window, the reference executes a query on the table in SAP

HANA and uses the returned data in the join to enrich streaming data with information from the database.

6.2 Renaming a Project

Change the streaming project name for easier recognition, or to reflect any changes in the content.

Context

From the SAP HANA Streaming Development perspective:

Procedure

1. Select the project folder from the project explorer field.

2. Right click and select Rename.

The same naming rules that apply when creating a project also apply when renaming a project:

○ Starts with a lowercase letter, underscore, or dollar sign.

○ Uses lowercase letters, numbers, underscores, or dollar signs for all other characters.

○ Cannot contain spaces.

3. In the Rename Resource field, enter the new project name.

All three of the files in the project folder are automatically renamed:

<project_name> .ccl

contains the CCL code.

<project_name> .cclnotation

<project_name> .ccr

contains the diagram that corresponds to the .ccl file.

contains the project configuration.

4. Click OK.

You will see the name change reflected in the project folder and the three project files.

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

6.3 Deleting a Project

Delete unused streaming projects from the smart data streaming plugin for SAP HANA studio.

Procedure

1. In the SAP HANA Streaming Development perspective, navigate to the project explorer and select a project, or multiple projects.

2. Right click and select Delete.

3. (Optional) In the dialog, click Preview to view a list of the selected projects. If you selected multiple projects to delete, the preview option allows you to uncheck any projects you want to keep.

4. (Optional) Check the Delete project contents on disk option to completely erase projects from the smart data streaming workspace and from your machine.

If this option is not checked, smart data streaming deletes the project from the project explorer, but does not remove the project from your workspace directory.

5. Click OK.

6.4 Adding an Adapter to a Project

Attach an adapter to a project by inserting it into the project diagram, connecting it to a stream or window, and setting properties.

Procedure

1. From the SAP HANA Streaming Development perspective, open the Input Adapters or Output Adapters compartment in the palette.

2. Click an adapter shape in the palette, then click in the diagram.

3. Attach the adapter to a stream or window. Either:

○ Generate and attach the stream or window automatically, using schema discovery (best practice for adapters that support it), or,

○ Create the stream or window, then attach it to an:

○ Input adapter: click the Connector tool, then click the adapter shape in the diagram, then click the stream or window.

○ Output adapter: click the Connector tool, then click the stream or window in the diagram, then click the adapter shape.

4. (Optional) Edit the adapter name.

5. (Optional) Edit the adapter properties. Choose one of the following:

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

○ Select Consolidate Adapter Properties from the adapter properties dialog. When this setting is enabled, the system looks for matching property pairings in the CCR, and removes them from the adapter properties collection. This setting is only relevant when an adapter has a property set configured; however, it is a global setting that affects all such adapters, not just the one you are currently attaching to the project.

○ Select Use named property set to use a named property set from the project configuration file, then configure any properties that are not included in the property set.

○ In the table, modify adapter properties manually.

Related Information

Discovering a Schema [page 98]

Adapters [page 40]

6.5 Working with Schemas

Discover external schemas and create CCL schemas, streams, or windows based on the format of the data from the data source that is connected to an adapter.

Rather than manually creating a new schema in your smart data streaming project, you can use schema discovery to discover and automatically create a schema, stream, or window based on the format of the data from the data source to which your adapter connects. For example, if you create a table in your SAP HANA database, use the SAP HANA Output adapter to connect to the database. You can then use schema discovery to discover and create a schema, stream, or window in your smart data streaming project that corresponds to the schema of the table you created in your SAP HANA database.

While using discovery is a convenient way to create your CCL schema, pay attention to the datatypes that the

CCL columns inherit from the external data source. Discovery tries to maintain the same level of precision, or greater, when mapping source datatypes to smart data streaming datatypes. Some databases, such as SAP

IQ, support microsecond precision for the SQL_TIMESTAMP and SQL_TYPE_TIMESTAMP datatypes. As such, schema discovery maps these types to the smart data streaming datatype bigdatetime, which also supports microsecond precision. If your smart data streaming project does not require this level of precision, you can, after generating your schema through discovery, modify the schema to use a lower-precision datatype, such as msdate for millisecond precision.

To enable schema discovery, configure the properties of the adapters that support the feature.

In this section:

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

Use the Schema Discovery button in the visual editor to discover and automatically create a schema based on the format of the data from the adapter.

Discovering Schema and Creating a Mapping File for the SAP RFC Adapter [page 100]

Use the Schema Discovery button in the visual editor to discover function, table, or query schema and create a mapping file for the RFC adapter.

Discovering Schema and Creating a Mapping File for the Web Services (SOAP) Adapter [page 102]

Use the Schema Discovery button in the visual editor to discover schema and create a mapping file for the Web Services (SOAP) adapter.

Creating a Schema in the Visual Editor [page 103]

Create a shared schema object that can be referenced from any number of streams or windows.

Creating a Schema in the CCL Editor [page 103]

Enter a CREATE SCHEMA statement using the CCL editor to provide users with a shared schema object that can be referenced from any number of streams or windows.

Adapter Support for Schema Discovery [page 104]

Some of the adapters that come with smart data streaming support schema discovery, while others do not. If you use an adapter that supports schema discovery, you need to set certain properties.

Related Information

Schemas [page 40]

SAP HANA Smart Data Streaming: Configuration and Administration Guide

6.5.1 Discovering a Schema

Use the Schema Discovery button in the visual editor to discover and automatically create a schema based on the format of the data from the adapter.

Prerequisites

● For a database adapter, you need access to the database on the system from which you are using SAP

HANA studio.

● Define a data service using the smart data streaming data services view, and add the adapter to the diagram.

● When using the SAP HANA smart data streaming plugin for SAP HANA studio, there is no default cluster.

Define a cluster in the server view of the SAP HANA Streaming Run-Test perspective before you can update the Default Server URL preference, or the server will not be available to select. See

Connecting to a

Cluster [page 87]

● Discovery uses the Default Server URL preference to retrieve the list of available data services. The default server URL is esp://localhost:9786. Because there is no localhost smart data streaming server, to locate available data services, update the Default Server URL preference:

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

1. In the SAP HANA Streaming Run-Test perspective, select Window Preferences .

2. In the Preferences dialog, select Smart Data Streaming.

3. In the Default Server URL field, click Change and select a server from the Select Default Server URL dialog. Click OK.

4. In the Preferences dialog, click Apply, then OK.

Procedure

1. In the SAP HANA Streaming Development perspective, open or create a project with an input adapter and configure the adapter for schema discovery. In the adapter shape, click Edit Properties and complete the dialog:

○ Select Use named property set and select a property set from the drop down menu, or,

○ Select Consolidate adapter properties from 'Adapter Properties' dialog and enter property values in the Basic and (optionally) Advanced tabs. Required properties are in red.

For example, to use schema discovery for the File CSV Input adapter, first configure the directory and file properties for the adapter, to specify the absolute path to the data files you want the adapter to read.

Note

To create a named property set, edit adapter properties in the project configuration file.

2. Click Schema Discovery on the adapter toolbar.

○ If the schema is successfully discovered, a dialog appears where you can view and select a schema.

○ If the schema is not successfully discovered, an error message appears stating that no schema was discovered for the adapter. You can:

○ Check that the adapter properties are configured for schema discovery.

○ Check to see if the adapter supports schema discovery.

3. Select a schema, and click Next.

4. In the dialog for creating an element, select an option:

Adapter State

The adapter is not attached to a stream or window.

Available Options

○ Create a new input stream (with inline schema): creates and attaches a new stream to the adapter, creates an inline schema for the stream, and populates the stream with the schema discovered from the adapter.

○ Create a new input window (with inline schema): creates and attaches a new window to the adapter, creates an inline schema for the window, and populates the window with the schema discovered from the adapter.

○ Create a new input stream (with attached schema): creates and attaches a new stream to the adapter, creates and attaches a new named schema to the stream, and populates the stream with the schema discovered from the adapter.

○ Create a new input window (with attached schema): creates and attaches a new window to the adapter, creates and attaches a new named schema to the window, and populates the window with the schema discovered from the adapter.

○ Create a new named schema: creates a new named schema and populates it with the schema discovered from the adapter.

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

Adapter State

The adapter is al ready attached to a stream or win dow.

Available Options

○ Apply the schema to the connecting stream or window: populates the stream or window with the schema discovered from the adapter.

○ Create a new named schema: creates a new named schema and populates it with the schema discovered from the adapter.

5. Click Finish.

○ The mapping file you specified in the Adapter Mapping File property is populated with mappings based on the schema you selected.

○ Either the window or stream that is attached to the adapter is populated with the schema you selected or a new named schema is created in the project to which the adapter is attached.

6. (Optional) Create or modify a mapping of adapter columns to smart data streaming columns.

a. In the adapter shape, click Edit Properties .

b. In the Adapter Properties screen, click the Advanced tab.

c. Click the Value column of the Field Mapping row.

The system displays an ellipsis in that field.

d. Click on the ellipsis.

The system displays the Define Adapter Field Mapping (Permutation) screen.

e. Click the Column field next to the database column you want to map to your smart data streaming column.

A down arrow is displayed, indicating a dropdown list of choices is available.

f. Click the mouse in the entry field to display the dropdown list and select the smart data streaming column to which you wish to map the database column.

g. Clear the check boxes next to any database columns that you do not wish to map (only checked columns will be mapped) and click OK.

Studio removes the dialog and redisplays the Adapter Properties screen with the new mapping in the

Field Mapping Value column.

h. You can also click Select All to place a check in all the database column check boxes or Remove All to remove the check from all of the check boxes.

6.5.2 Discovering Schema and Creating a Mapping File for the SAP RFC Adapter

Use the Schema Discovery button in the visual editor to discover function, table, or query schema and create a mapping file for the RFC adapter.

Prerequisites

Add the SAP RFC adapter to the diagram.

100

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

Procedure

1. In the SAP HANA Streaming Development perspective, configure the RFC Input or Output adapter for schema discovery. In the adapter shape, click Edit Properties and set these properties:

○ Adapter Configuration File

○ Adapter Mapping File

○ SAP Host

○ SAP System Number

○ SAP Client

○ Username

○ Password

Ensure there are no checkboxes selected, and click OK.

2. Click Schema Discovery on the adapter toolbar.

○ If the schema is successfully discovered, a dialog appears where you can view and select schemas.

○ If the schema is not successfully discovered, an error message appears stating that no schema was discovered for the adapter. Check that the adapter properties are configured for schema discovery and that no checkboxes are selected in the Edit adapter properties dialog.

3. In the schema discovery dialog:

○ (RFC Input adapter only) Select which remote schemas to search: Functions, Tables, or Queries.

Scroll through to view the discovered functions, tables, or queries.

○ (RFC Output adapter only) Scroll through to view the discovered schemas. Only function schemas display.

4. Select a function, table, or query and click Select Schema.

5. Click Next.

6. In the Create Element dialog, select an option:

○ Assign schema to the connecting stream/window: populates the stream or window to which the adapter is attached with the selected adapter schema.

○ Create new named schema: creates a new named schema in smart data streaming which is made up of the RFC schema elements.

7. Click Next.

8. From the left column, select the remote fields for your schema and click Select Field(s).

These fields now appear under the Selected Mapping fields column. To remove any fields from this column, select the field and click Select Field(s) again.

9. Click Finish.

○ The mapping file you specified in the Adapter Mapping File property is populated with mappings based on the schema you selected.

○ Either the window or stream that is attached to the adapter is populated with the schema you selected or a new named schema is created in the project to which the adapter is attached.

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

101

Next Steps

● For BW mode, edit the generated mapping file by adding the <variables> element.

● For Generic RFC mode, edit the generated mapping file by adding the <variables> and <input> elements.

6.5.3 Discovering Schema and Creating a Mapping File for the Web Services (SOAP) Adapter

Use the Schema Discovery button in the visual editor to discover schema and create a mapping file for the

Web Services (SOAP) adapter.

Prerequisites

Add the Web Services (SOAP) adapter to the diagram.

Procedure

1. In the SAP HANA Streaming Development perspective, configure the Web Services (SOAP) Input or

Output adapter for schema discovery. In the adapter shape, click Edit Properties and set these properties:

○ Adapter Configuration File

○ Adapter Mapping File

○ Discovery WSDL URL

○ Discovery Working Directory

○ Discovery Service Name

Ensure there are no checkboxes selected, and click OK.

2. Click Schema Discovery on the adapter toolbar.

○ If the schema is successfully discovered, a dialog appears where you can view and select schemas.

3. Select a discovered schema and click Next.

4. In the Create Element dialog, select an option:

○ Assign schema to the connecting stream/window: this populates the stream or window to which the adapter is attached with the selected adapter schema.

○ Create new named schema: this creates a new named schema in smart data streaming which is made up of the Web Services (SOAP) schema elements.

102

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

5. Click Finish.

○ The mapping file you specified in the Adapter Mapping File property is populated with mappings based on the schema you selected.

○ Either the window or stream that is attached to the adapter is populated with the schema you selected or a new named schema is created in the project to which the adapter is attached.

6.5.4 Creating a Schema in the Visual Editor

Create a shared schema object that can be referenced from any number of streams or windows.

Procedure

1. From the SAP HANA Streaming Development perspective, in the palette menu under the Shared

Components category, select Named Schema ( ).

2. Click anywhere in the visual editor to place the schema.

3. Set the name of the schema by either:

○ Double-clicking the name label, or,

○ Editing the name field from within the Properties window.

4. Click Add Columns ( ) to add individual columns.

5. Edit column names and datatypes.

Tip

When entering column names and their datatypes, use Tab to easily move between cells in the table.

6. (Optional) Connect the schema to one or more streams or windows using the connector tool.

6.5.5 Creating a Schema in the CCL Editor

Enter a CREATE SCHEMA statement using the CCL editor to provide users with a shared schema object that can be referenced from any number of streams or windows.

Procedure

In the CCL editor, enter valid CCL for the CREATE SCHEMA statement.

○ Enter text manually.

○ Choose Create Schema , and edit the draft CCL code as needed.

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

103

For example, this statement creates a shared schema object named SchemaTrades1, with four columns:

CREATE SCHEMA SchemaTrades1 (

Symbol STRING ,

Seller STRING ,

Buyer STRING ,

Price FLOAT )

6.5.6 Adapter Support for Schema Discovery

Some of the adapters that come with smart data streaming support schema discovery, while others do not. If you use an adapter that supports schema discovery, you need to set certain properties.

For additional details on the adapter properties, see the specific adapter section.

Adapter

Database Input

Supports Schema Discovery

Yes

Required Properties

Database Service

Name of database service from which the adapter obtains the database connection.

Database Output Yes

Database Service

Name of service entry to use.

File/Hadoop CSV Input Yes

File/Hadoop CSV Output

File/Hadoop JSON Input

File/Hadoop JSON Output

File/Hadoop XML Input

File/Hadoop XML Output

File/Hadoop XML Event Input

Yes

—

File Input Transporter:

● Dir

● File

● AccessMode

● (Optional) ScanDepth

CSV String to Streaming Formatter

● ExpectStreamNameOpcode

File/Hadoop XML Event Output No

File Input Transporter:

● Dir

● File

● AccessMode

● (Optional) ScanDepth

—

104

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

Adapter

File FIX Input

File FIX Output

FTP CSV Input

FTP CSV Output

FTP Event XML Input

FTP Event XML Output

HTTP Client Output

HTTP Output

JDBC Input

JDBC Output

JMS CSV Input

JMS CSV Output

Yes

Supports Schema Discovery

Yes

—

Required Properties

—

JDBC Input Transporter:

● Host

● Port

● User

● Password

● DbName

● DbType

● DbDriver

JDBC Output Transporter:

● Host

● Port

● User

● Password

● DbName

● DbType

● DbDriver

JMS Input Transporter:

● ConnectionFactory

● JndiContextFactory

● JndiURL

● DestinationType

● DestinationName

● MessageType

● (Optional) ScanDepth

—

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

105

Adapter

JMS Event XML Input

JMS Event XML Output

JMS FIX Input

JMS FIX Output

JMS MAP Input

JMS MAP Output

JMS Object Input

JMS Object Output

Log File Input

Random Tuples Generator Input No

Replication Server Input Yes

SAP ASE Output

SAP HANA Output

SAP IQ Output

Supports Schema Discovery

Yes

Required Properties

JMS Input Transporter:

● ConnectionFactory

● JndiContextFactory

● JndiURL

● DestinationType

● DestinationName

● MessageType

● (Optional) ScanDepth

—

● RSSD Host

● RSSD Port

● RSSD Database Name

● RSSD User Name

● RSSD Password

DB Service Name

The name of the database service that repre sents the SAP ASE database into which infor mation will be loaded.

DB Service Name

The name of the database service that repre sents the SAP HANA database into which in formation will be loaded.

DB Service Name

The name of the database service that repre sents the IQ database into which information will be loaded.

106

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

Adapter

SAP RFC Input

SAP RFC Output

SMTP Output

Socket CSV Input

Socket CSV Output

Socket Event XML Input

Socket Event XML Output

Socket FIX Input

Socket FIX Output

Socket JSON Input

Socket JSON Output

Tibco Rendezvous Input

Tibco Rendezvous Output

Web Services (SOAP) Input

Web Services (SOAP) Output

WebSphere MQ Input

WebSphere MQ Output

Supports Schema Discovery

Yes

Required Properties

● Adapter Configuration File

● Adapter Mapping File

● SAP Host

● SAP System Number

● SAP Client

● Username

● Password

—

● Adapter Configuration File

● Adapter Mapping File

● SAP Host

● SAP System Number

● SAP Client

● Username

● Password

● Adapter Configuration File

● Adapter Mapping File

● Discovery WSDL URL

● Discovery Working Directory

● Discovery Service Name

—

● Adapter Configuration File

● Adapter Mapping File

● Discovery WSDL URL

● Discovery Working Directory

● Discovery Service Name

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

107

Adapter Supports Schema Discovery

Streaming Web Services Provider No

Streaming Web Output No

Required Properties

—

6.6 Adding an Input Stream or Window to a Project

You can add input streams and windows to a project, which accept data from a source external to the streaming project.

Context

You can create an input stream or window by adding an adapter that supports schema discovery, and generating the stream or window to inherit the schema of the external data source automatically. You can then add columns, and specify whether they need an autogenerate clause, which automatically generates data for specified columns.

Procedure

1. From the SAP HANA Streaming Development perspective, in the palette menu of the visual editor, open the Streams and Windows category and select either:

○ Input Stream

○ Input Window

2. Select a location in the diagram and click to add the shape.

3. To set the name of the input stream or window, either:

○ Click to edit the shape name, or,

○ In verbose mode, click the Edit icon next to the name.

Note

When you create a duplicate named window or stream in the visual editor, then save your file and switch to the CCL editor, a third copy of the original stream or window is created. You can see this third copy only when you switch back to the visual editor. To remove it, click Remove all shapes from diagram to clear out all the shapes, then click Add all shapes to bring back the original stream or window, and the duplicate stream or window.

4. Click Add Column to add each new column to the schema, then set key columns and edit column expressions.

108

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

5. (Optional) To delete columns, select them and press Delete.

6. (Optional for windows, not permitted for streams) Select Set Keep Policy and choose an option.

7. (Optional) Double-click the policy to edit its parameters.

8. (Optional for both windows and streams) Select Set Autogenerate , choose the columns from the

Candidate list (only columns with a long datatype will populate the Candidate list) and click Add.

Note

You can also manually specify a column to add to the autogenerate list by clicking Add Column and entering in a column name. You can only use columns with the long datatype.

9. (Optional) To remove columns from the autogenerate list, select them and click Remove.

10. To set a From value for the autogenerate clause to start with, click Select and choose a variable or parameter from the list. You can also manually enter a variable or parameter that is used within a declare block of a column with a long datatype.

11. Click OK.

6.7 Specifying a Retention Policy

The keep policy determines the basis for retaining rows in a window.

You can set a keep (or retention) policy for any window with a memory-based store, including any simple query that produces a window.

Retention policies for windows that use a log store are only supported for input windows.

The following table describes the options for keep policies:

Options

All rows

Last row

Description

Retain all rows in the window (default).

Retain only the last row in the window.

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

109

Options

Count

Every

Slack

Time

PER clause

Description

Either:

● Enter the absolute number of rows to retain, or,

● Choose Select and select a previously declared variable or parameter to determine a specific range of rows to retain in the window.

Tip

If the list is empty and you want to base the count on a parameter or variable, switch to the CCL editor and define it in a DECLARE block at the beginning of the

CCL. For example:

DECLARE integer test :=50; end;

Then go back and select it.

(Optional) Works with the Count and Time options.

When used with the Count option, Every retains a number of rows based on the Count value specified, and purges all of the retained rows once a row arrives that would exceed the specified maximum number of rows. This purge occurs after the specified Count number has been reached.

When used with the Time option, Every retains a number of rows within a specified time interval. Once the time interval expires, all rows are purged simultaneously.

Note

When this option is used, the resulting retention is based on a jumping window pol icy. Otherwise, the resulting retention is based on a sliding window policy.

For a count-based policy, set the number of rows to delete when the maximum number of rows is reached (the Count value). Default is 1, that is, when the window contains

<count-value>

rows, each new row causes the oldest row to be deleted. Setting Slack to greater than 1 can optimize performance.

Set a time limit on the window, and specify a time period to determine what age of row to retain in the window. Press Ctrl+Space to choose the unit of time.

(Optional) Works with the Time and Count options.

When used with the Count option, PER works in conjunction with the specified Count number to retain the Count number of rows across each column specified under the

PER clause.

When used with the Time option, PER works in conjunction with the specified Time in terval to retain the rows within that Time interval across each column specified under the PER clause.

List the names of the columns that need to be retained in the PER clause box, with a comma separating each column name entered.

110

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

Count

In a sliding window count-based retention policy, a constant integer specifies the maximum number of rows retained in the window. To retain the specified maximum number of rows in the window, the policy purges the oldest entries as new entries arrive, one row at a time.

In a jumping window count-based retention policy, enabled by using the Every option, all rows are purged only once a row arrives that would exceed the specified maximum number of rows.

A sliding window count-based policy also defines an optional slack value, which can enhance performance by requiring less frequent cleaning of memory stores.

Slack

Slack is an advanced feature used to enhance performance by requiring less frequent cleaning of memory stores. It sets a maximum of N + S rows in the window, where N is the retention size (the count setting) and S is the slack. When the window reaches N + S rows, the systems purges S rows. The larger the value of slack the better the performance is, since there is less cleaning required.

The default value for Slack is 1. When slack = 1, after the window reaches the maximum number of records, each time a new record is inserted, the oldest record is deleted. This causes a significant impact on performance. When slack > 1, say Y, then the window will accumulate up to X + Y number of records. The next record inserted will then cause the deletion of Y records. Larger slack values improve performance by reducing the need to constantly delete rows.

Note

You cannot use slack with the Every option.

Time

In a sliding window time-based retention policy, a time interval specifies the maximum age of the rows retained in the window. Rows are purged from the window, one row at a time, when they become older than the specified interval.

In a jumping window time-based retention policy, enabled by using the Every option, all rows produced in the specified time interval are purged after the interval has expired.

PER Clause

The PER clause allows for rows specified by the Count or Time options to be retained across specified columns.

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

111

For a count-based retention policy, it keeps the number of rows specified by the Count number across each column specified under the PER clause. The rows in each column specified to be retained will update simultaneously to delete older entries as newer ones arrive.

For a time-based retention policy, it keeps rows within the specified Time interval across each column specified under the PER clause. The rows in each column specified to be retained will update simultaneously to delete older entries as the time interval expires.

Related Information

Streams and Windows [page 22]

Creating and Modifying Aggregate Queries [page 163]

6.8 Connecting Elements

Connect two shapes in a diagram to create a data flow or link between them.

Context

The Connector tool creates flows between streams and windows, establishes references between streams and shared components, or attaches notes between shapes.

Procedure

1. From the SAP HANA Streaming Development perspective visual editor, in the palette view, select the

Connector tool.

2. Click the shape that will produce the output.

This attaches the connector line to the first shape.

3. Click the shape that will receive the data to indicate the direction of data flow.

Tip

To add multiple connections, Shift+click while holding the Connector tool. To return to normal selection, press Esc or click the Select tool in the palette to release it.

112

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

Related Information

Shape Reference [page 68]

6.9 Setting Key Columns

Set primary keys in the visual editor within the column compartment of the window, and Flex operator shapes.

Context

Multiple columns can be designated as primary keys. In the visual editor, primary keys appear as icons.

Deduced primary keys, which appear as icons, are calculated when the PRIMARY KEY DEDUCED flag is set for the target element.

Note

Only windows support PRIMARY KEY DEDUCED. You can modify the deduced key property for these elements from the Properties view.

The ability to set key columns and view key column icons as described here does not apply when using column expressions.

Procedure

1. From the SAP HANA Streaming Development perspective visual editor, expand the Columns compartment of the desired query object (window, or Flex shape).

2. Click the icon to the left of the column name to make it a primary key.

A single-key icon now designates the column as a primary key.

3. (Optional) To set a primary key for query objects with a deduced primary key, click any column or deduced key within the target window.

The column you initially selected and all other deduced key columns are now primary keys. In addition, the target window is no longer PRIMARY KEY DEDUCED.

4. (Optional) To remove the primary key designation from a column, click to the left of the column name.

A column icon replaces the single key icon, indicating that the column is no longer part of the primary key.

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

113

6.10 Editing Column Expressions

Modify column expressions for windows and streams using an inline editor or dialog-based expression editor.

Procedure

1. (Optional) To add a column expression, click Add Column Expressions in the shape toolbar.

2. Expand the Column Expressions compartment.

3. To modify a column expression, choose one of the following options:

○ Double-click to open the inline editor. Type into the edit box to edit the existing expression or enter a new one. Press Ctrl+Space for a list of available columns and functions.

○ Press Ctrl+F2 to open the expression editor. Press Ctrl+Space to show the available input columns and built-in functions, or manually enter the expression.

○ Modify the expression in the Properties view.

Tip

When entering column names and their datatypes, use Tab to easily move between cells in the table.

Related Information

CCL Functions [page 76]

Operators [page 49]

6.11 Deleting an Element

Delete an element to remove it from the project completely, or delete it from the diagram while retaining it in the project.

Procedure

1. Select one or more elements in the diagram.

2. Right-click and choose either:

○ Delete Element: removes the element from the project.

○ Delete from Diagram: removes the element from the diagram, but retains it in the project. When you run the project, everything in the project runs, including elements that are not in the diagram.

114

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

3. When you choose Delete Element, confirm the deletion.

6.12 Creating Windows and Streams from a Database Table

Create windows and streams from SAP HANA tables for a smart data streaming project in studio.

Procedure

1. In SAP HANA studio, open the SAP HANA Streaming Development perspective, and select the systems view or data services view.

2. Select an SAP HANA table.

3. Drag and drop the SAP HANA table into the project workspace.

4. From the Create CCL from HANA Table dialog, click the Service button, and select a data service.

5. From Create Type, specify the type of stream or window to create:

○ Input Stream

○ Output Stream

○ Input Window

○ Output Window

6. If you selected output stream or output window in the previous step, specify the Output Adapter Type as:

○ Generic Database Output (default)

○ Hana Database Output

7. Select a Schema:

○ Named Schema Assignment

○ Inline

8. Click OK.

6.13 Using the SAP HANA Repository

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

115

Procedure

1. Create a new smart data streaming project in the SAP HANA studio.

2. Share the project to create a link between project-specific files in your development workspace and the

SAP HANA repository. See Tutorial: Share an Application Project in the SAP HANA Developer Guide .

Saving the project in your workspace automatically saves it to the repository.

3. Compile your project when you are ready to test it.

At this point, you can create a delivery unit. See Create a Delivery Unit in the SAP HANA Developer Guide .

4. Right-click on your project folder, select Team Activate to activate, and automatically deploy your project to the smart data streaming server.

Results

Switch to the SAP HANA Streaming Run-Test perspective to verify that the project is running.

If the project is not running after one minute and does not appear in the SAP HANA Streaming Run-Test perspective, use the XS Job Dashboard in the SAP HANA XS Administration Tool to locate the sap.hana.streaming.repoplugin::plugin.xsjob job and troubleshoot errors. See The XS Job

Dashboard in the SAP HANA Administration Guide for additional details. Make any necessary changes to your project CCL file, then recompile and reactivate the project.

Related Information

SAP HANA Developer Guide

SAP HANA Administration Guide

6.14 Compiling a Streaming Project

When you compile a project, smart data streaming produces an executable .ccx file from CCL code. This file is necessary to run a project on smart data streaming.

Procedure

1. In the SAP HANA Streaming Development perspective, in the project explorer, expand the tree view to show the .ccl file for the project.

2. Select and open the .ccl project you are compiling.

116

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

3. To compile a project without running it, either to check for errors or just to have an updated .ccx file, click

Compile Project on the main toolbar, or press F7.

4. To compile and run the project, click Run Streaming Project .

The project automatically compiles and runs. The server view in the SAP HANA Streaming Run-Test perspective opens, showing the project connection. A successful connection displays the server streams below the server folder. If the connection is unsuccessful, you see a server connection error dialog.

Results

Studio silently saves all open files belonging to the project, compiles the project, and creates the .ccx file (the compiled executable). Compilation errors are displayed in Problems or Console view in each perspective, depending on the type of error. If you selected Run Project , it also runs the compiled project.

Note

The .ccx file is platform-independent. You can migrate, test, develop, and run the file in a studio running on different operating systems. For example, you can compile a project in studio on Unix, and deploy the same file in studio on Windows.

Studio returns an error when a project refers to a schema from an imported file but the project compiles without errors. Refresh the file by closing the project or create the files in the opposite order.

6.15 Compiling Projects When Running SAP HANA Studio on

Mac OS

Before you can compile and run a streaming project in SAP HANA studio on Mac OS, or on Linux with no smart data streaming server installed, first connect to an SAP HANA smart data streaming server.

Prerequisites

When using smart data streaming in SAP HANA studio on Mac, or on Linux with no smart data streaming server installed, there is no default cluster so you need to update the Default Server URL preference. Before you can update this URL, connect to an SAP HANA smart data streaming cluster in the server view of the SAP

HANA Streaming Run-Test perspective. Otherwise, the SAP HANA smart data streaming server will not be

available to select. See Connecting to a Cluster [page 87]

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

P U B L I C

117

Procedure

1. In the SAP HANA Streaming Run-Test perspective of SAP HANA studio, select Window

Preferences .

2. In the Preferences dialog, select SAP HANA smart data streaming.

3. In the Default Server URL field, click Change and select a server from the Select Default Server URL dialog. Click OK.

4. In the Preferences dialog, click Apply, then OK.

5. (Optional) To compile a project without running it, click Compile Project on the main toolbar or press

F7.

6. (Optional) To compile a project and run it, click Run Streaming Project .

Results

SAP HANA studio is now connected to an SAP HANA smart data streaming server.

118

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing Streaming Projects

7 Running and Testing Streaming Projects

Test a streaming project by compiling and running it on a server, accessing and filtering streams, saving and uploading data to the SAP HANA smart data streaming server, and setting project configurations.

In this section:

Accessing the Run-Test Perspective [page 120]

Open the SAP HANA Streaming Run-Test perspective for toolbars and views that simplify testing, monitoring, debugging, and examining smart data streaming projects.

Caching Project Settings [page 120]

When running projects within studio, you can save time by caching your run-test views so they open to your previous settings.

Running a Streaming Project [page 121]

To run a streaming project in SAP HANA studio on Windows or Linux, first connect to an SAP HANA smart data streaming server. Once connected, run the project to upload data from a file, manually enter data, and view streams in that project.

Monitoring Project Performance [page 126]

The Monitor View shows visual indicators of queue size, throughput, and CPU use for each stream and window (including local streams and windows) in a project.

Running a Snapshot SQL Query against a Window [page 128]

In the SQL Query view, run a snapshot SQL query against an output window in a running project, and show the results in the Console.

Playback View [page 129]

The playback view records incoming data to a playback file, and plays the captured data back into a running smart data streaming instance. You can also use it in place of the file upload tool to upload data files at a controlled rate. All date and time stamps within the playback view are assumed to be in

UTC.

Debugging [page 134]

The SAP HANA Streaming Run-Test perspective in studio contains two tools for debugging data flow and to assist you in locating and fixing bugs within the project: the debugger, which allows you to set breakpoints, and the event tracer, which shows the impact of each incoming event on all streams and windows of a project.

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

119

7.1 Accessing the Run-Test Perspective

Open the SAP HANA Streaming Run-Test perspective for toolbars and views that simplify testing, monitoring, debugging, and examining smart data streaming projects.

Procedure

Click the SAP HANA Streaming Run-Test tab at the top of the studio main window to see the SAP HANA

Streaming Run-Test perspective.

If the tab is not visible, from the main menu select Window Open Perspective Other SAP HANA

Streaming Run-Test .

7.2 Caching Project Settings

When running projects within studio, you can save time by caching your run-test views so they open to your previous settings.

Context

Items cached include:

● The list of Stream View stream tabs.

● The current Manual Input Stream.

● The current SQL Query, Playback, Upload, Monitor, Event Tracer projects.

Procedure

1. Click Windows Preferences Smart Data Streaming Run-Test .

2. Select Always start previously running run-test views on project start.

3. Uncheck Clear all cached run-test views.

Note

Ensure that you clear this option, because if you clear the cache, there will not be anything saved to load. Select this option only if you do not want to cache your run-test settings.

120

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

7.3 Running a Streaming Project

Prerequisites

● To run a streaming project, ensure that one or more connected workspaces are available.

● When studio is installed on a separate machine from the streaming server, you cannot run projects on the default local cluster. Instead, connect to the streaming server to run your project.

● Connect to SAP HANA smart data streaming server. See

Connecting to a Cluster [page 87] for detailed

instructions.

Procedure

1. Select and open the .ccl file you want to run.

If no editors are open, pick a project to run.

2. To run the project, either:

○ Click Run Project

in the main toolbar (in either the SAP HANA Streaming Development or the SAP

HANA Streaming Run-Test perspective) to run the project in the default/most recently used workspace, or,

○ Click the dropdown arrow next to the Run Project tool and choose Run Project in Workspace, then select the workspace where this project will run.

The project runs and shows results in the SAP HANA Streaming Run-Test perspective.

Note

If you have run the project before and encounter an error stating that a failure occurred because the application already exists, the project may still exist on the server in a stopped state. Remove the project from the SAP HANA Streaming Run-Test server view, and restart the project from the SAP

HANA Streaming Development perspective.

In this section:

Working in Server View [page 122]

The server view shows SAP HANA smart data streaming servers available for connecting and running projects.

Viewing a Stream [page 122]

Stream view shows all of the events of an output stream and all of the retained events in an output window for the running project.

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

121

Specifying a Pulse Rate [page 123]

You can specify how often you see changes and updates in a stream. When a data stream contains few items with a high volume of changes, you can set a pulse rate so that changes are delivered periodically, in optimally coalesced blocks.

Uploading Data to the SAP HANA Smart Data Streaming Server [page 124]

Use the File Upload tool to load event data from files into a running project. Normally, you use this tool to test projects. Date and time stamps in data loaded through the File Upload tool are assumed to be in the local timezone.

Manually Entering Data to a Stream [page 125]

Manually create and publish an event as input to a stream or window. By default, date and time stamps in data loaded through the Manual Input tool are assumed to be in the local timezone. You can change this setting to use Universal Coordinated Time (UTC) through your Studio preferences.

7.3.1 Working in Server View

The server view shows SAP HANA smart data streaming servers available for connecting and running projects.

You can:

● Connect a project.

● Add a new server URL to the list of available connections, remove an existing server, or reconnect all listed servers.

● Show a server in monitor view or event tracer view.

● Load projects into a workspace.

7.3.2 Viewing a Stream

Stream view shows all of the events of an output stream and all of the retained events in an output window for the running project.

Procedure

1. In the SAP HANA Streaming Run-Test perspective, select the stream or window from the server view.

2. Right-click the output stream or window, and select Show In StreamViewer (or New

StreamViewer).

A tab opens in the stream view showing all new events. If you selected a window, all retained rows currently in the window are displayed.

3. To manipulate your subscription list, or individual stream subscriptions, select the subscription to edit and choose one of these buttons at the top of the stream view:

○ Close Subscription URL

disconnects and closes the stream view.

○ Clear

clears contents and pauses the subscription.

122

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

○ Show Current Subscription in new View

shows the publish date of the stream (if available).

○ Set StreamViewer number of rows displayed

lets you choose the number of rows to show in the stream view. The number can be between 1 and 1000000. The default value is 25.

4. (Optional) To save data from the stream view, click Clipboard Copy .

7.3.3 Specifying a Pulse Rate

Context

A stream containing three ticker symbols, for example, may generate thousands of updates every second. You can set the pulse period to control the frequency at which you receive updates when viewing the stream. If you set the pulse to refresh every 5 seconds, the subscription then delivers, at most, one updated record for each of the three symbols every five seconds.

There are two preferences that control the subscription feature in studio:

● StreamViewer pulsed subscribe interval

● Other pulsed subscribe interval

Both preferences are measured in seconds. If either of these preferences is set to 0, then studio does not perform a pulsed subscription on the related stream.

Note

If you have a small data set and you set the pulse to refresh frequently, such as once every 1 or 2 seconds, the Stream View may be empty for some streams because there are no new updates

Go through the following steps to change the default settings:

Procedure

1. Choose Window Preferences .

2. In the left pane, expand Smart Data Streaming Run-Test .

3. Enter new values for StreamViewer pulsed subscribe interval or Other pulsed subscribe interval or both.

4. Click Apply.

5. Click OK to close the dialog.

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

123

7.3.4 Uploading Data to the SAP HANA Smart Data

Streaming Server

Prerequisites

Ensure that the project is running.

Procedure

1. In the SAP HANA Streaming Run-Test perspective, select the File Upload view in the lower-left pane.

Note

The File Upload tool uploads the data file as fast as possible. For playing back data at controlled rates, use the Playback tool.

2. Click Select Project in the toolbar in the upper right corner of the File Upload view.

3. Select the project to which you want data uploaded, and click OK.

4. Click Browse to open the file choice dialog and navigate to the input file to upload.

5. Select one or more files to upload:

○ The smart data streaming server supports smart data streaming binary (.bin), smart data streaming

XML (.xml), and comma-separated values and text (.csv or .txt) files. Regardless of file type, each record in the file has to start with the input stream or window name in the first field, followed by the opcode in the second field, then the actual contents of the record in the remaining fields.

○ Use the format yyyy-MM-'dd'T'HH:mm:ss when uploading any SecondDate data from a file. For example, TradeTime="2000-05-04T12:00:00".

○ Use the format yyyy-MM-'dd'T'HH:mm:ss.SSS when uploading any MsDate data from a file. For example, Tradetime="2000-05-04T12:00:00.000".

6. Click Upload. A progress bar tracks the upload status.

The File Upload view allows you to perform these additional actions:

UI control

Remove File

Cancel

Action

Discard a previously selected file from the Input File(s) menu.

Cancel a file upload currently in progress.

124

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

UI control Action

Note

Any data sent before the upload is cancelled is still processed.

Use Transaction Process multiple records as a single transaction. If Record Buffer is specified, group that many re cords in each transaction. If not, process the entire file as one transaction.

Record Buffer Specify the number of records to group together and process in a single transaction.

7.3.5 Manually Entering Data to a Stream

Context

Manually publishing input events to a project is useful when testing a project.

Procedure

1. In the SAP HANA Streaming Run-Test perspective, select the Manual Input view in the lower-left pane.

2. Click Select Stream ( ) in the toolbar in the upper right corner of the Manual Input view.

3. In the Select Stream dialog, select the stream and click OK.

4. Edit available data columns as desired.

5. To edit more than one row of the data, select Edit Multiple Rows ( ) and choose the rows to modify.

6. If you are publishing to a window, indicate the opcode by selecting one of the data events. If you are publishing to a stream, only insert events are supported.

7. (Optional) Select Use Current Date to change the value of any bigdatetime or date object in the manual input view to the present date.

8. Click Publish to send the event to the project.

Related Information

Manual Input Settings [page 81]

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

125

7.4 Monitoring Project Performance

The Monitor View shows visual indicators of queue size, throughput, and CPU use for each stream and window

(including local streams and windows) in a project.

Each node corresponds to a stream in the model with the lines outlining the path the data flows through. The color of each node represents either QueueDepth or Rows Processed (/sec), depending on your specifications.

For example, if you select the Color Queue Depth option, the (Red) Range >= field defaults to 125, and the

(Yellow) Range >= field defaults to 20. This means:

● If the queue depth of the stream node is greater than or equal to 125, the node is red.

● If the queue depth of the stream node is between 20 and 124, the node is yellow.

● If the queue depth of the stream node is less than 20, the node is green.

● If the nodes remain white, it indicates that the monitor is not receiving data from the stream processor.

The Monitor View depicts CPU utilization as a black pie wedge in the ellipses of the node. Based on the options chosen, the remainder of the ellipses are red, yellow or green. A fully black node represents 100 percent CPU use, based on a single CPU core. With multicore or multiprocessor environments, a fully black node may be greater than 100 percent.

You can look at a specific node's performance statistics by moving your cursor over the node in the diagram.

Note

Monitoring data is available only if the time-granularity option (otherwise known as the performance monitor refresh interval) is enabled. The time-granularity option specifies, in seconds, how often the set of performance records — one per stream and one per gateway connection — is obtained from the running smart data streaming project.

In this section:

Running the Performance Monitor [page 126]

View visual indicators of queue size and CPU usage for each stream and window.

Saving a Performance Diagram as an Image [page 128]

Save a performance diagram.

7.4.1 Running the Performance Monitor

View visual indicators of queue size and CPU usage for each stream and window.

Prerequisites

The project you wish to monitor is running.

126

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

Context

You can specify a delay by changing the performance timer interval.

Procedure

1. In studio SAP HANA Streaming Run-Test perspective, select the server view.

2. Right-click on a running project and select Project Properties.

3. In the dialog, set Performance Monitoring Refresh Interval to a number greater than 0.

The default setting is 5. The performance monitor will not run if the refresh interval is set to 0.

4. In the monitor view, click Select Running Project ( ).

5. Click OK.

6. Select QueueDepth or Rows Processed to specify how to color each node in the performance diagram.

For either option:

○ Type in a number or use the arrow buttons in the (Red) Range >= field to select the range to create a red node.

○ Type in a number or use the arrow buttons in the (Yellow) Range >= field to select the range to create a yellow node.

Note

Nodes are green when they fall within a range that is not in either the (Red) Range >= or the (Yellow)

Range >=.

7. Click Zoom In or Zoom Out to see a larger or smaller view of the diagram.

Task overview:

Monitoring Project Performance [page 126]

Related Information

Saving a Performance Diagram as an Image [page 128]

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

127

7.4.2 Saving a Performance Diagram as an Image

Save a performance diagram.

Context

You can modify the performance diagram properties using the Monitor window in the SAP HANA Streaming

Run-Test perspective. The diagram appears in the Event Tracer window, and can be saved only from that window.

Procedure

1. In the SAP HANA Streaming Run-Test perspective, select the Event Tracer view.

2. Click Save.

3. Enter a file name and save location. Click Save.

The file is saved as a JPEG image in the location you specified.

Task overview:

Monitoring Project Performance [page 126]

Related Information

Running the Performance Monitor [page 126]

7.5 Running a Snapshot SQL Query against a Window

In the SQL Query view, run a snapshot SQL query against an output window in a running project, and show the results in the Console.

Procedure

1. In the SAP HANA Streaming Run-Test perspective, select the SQL Query view in the lower-left pane.

2. Click Select Project .

3. Select the project and window to query, and click OK.

128

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

4. Enter your query.

For example,

Select * from <stream>

5. Click Execute.

The results are displayed in the Console.

7.6 Playback View

Playback View Options

Feature

Select playback file

Start playback

Stop playback

Start Recording

At Timestamp rate

XML/CSV datemask

Description

Select the file format to use with the smart data streaming recorder.

Starts playing the current playback file.

Stops playback or record, closes the associated file and closes the associated playback or record context.

Prompts the user to select the file in which to store recorded data and starts the smart data streaming recorder.

This slider is used during playback to vary the rate of playback

Applies a datetime mask to read data from the source file. The default datetime format is UTC, or %Y-%m-%dT%H:%M:%S.

This option cannot change delimiters, which are the characters that separate each value: "-", "T", ":").

Playback Mode Options

Feature

Full rate

Description

Full rate indicates that the speed of playback is not imposed by the studio. Full rate is dependent on factors such as the computer that is running studio, or net work latency.

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

129

Feature

Timestamp column

Rec/ms

Description

The timestamp column option tells the recorded file to play back using the tim ing rate information from a specified column. During playback, timestamps de termine the time interval between records.

To change the project runtime to the time the data was recorded:

1. Select timestamp column.

2. Check Use Recorded time.

3. Enter a column name in the timestamp column field.

To return to current time, restart the project.

The timestamp column field supports these datetime datatypes:

●

BIGDATETIME

SECONDDATE

TIME

INTERVAL

INTEGER

LONG

When entering a datemask in the XML/CSV datemask field, these datatypes have a default format:

●

BIGDATETIME : YYYY-MM-DDTHH:MM:SS:SSSSSS

SECONDDATE : YYYY-MM-DDTHH:MM:SS

TIME : HH24:MM:SS

Note

These formats cannot change delimiters.

These datatypes do not support a datemask:

●

INTERVAL

INTEGER

LONG

The records-per-millisecond (rec/ms) mode lets playback occur at a recordsper-millisecond rate. The option allows you to set an initial rec/ms rate, which you can modify using the At Timestamp rate slider tool.

Supported File Extensions and Operation Codes

The studio recorder supports these file formats:

● .xml (smart data streaming XML)

● .csv (comma-separated values)

● .bin (smart data streaming Binary)

130

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

● .rec (studio recorded file)

Regardless of file type, each record in the file has to start with the input stream or window name in the first field, followed by the opcode in the second field, then the actual contents of the record in the remaining fields.

The default date/time format for all supported date/time datatypes is %Y-%m-%dT%H:%M:%S. For example,

2011-06-30T17:00:00.000.

Supported operation codes, and their abbreviated forms, are:

● INSERT: I or i

● UPDATE: U or u

● DELETE: D or d

● SAFEDELETE: S or s

● UPSERT: P or p

Additional Formatting Rules for CSV and XML Files

For CSV files, the value order has to correspond to the schema definition of the input stream or window. For example: isSensorStream,i,2011-06-30T17:00:00.000,34.614,-111.843,Tower,Camp Verde

T11,1,96.8

For CSV files, null values are only allowed if the correct number of fields are represented and delimited by commas (or whatever single-character delimiter is used from the ASCII printable character set). For example: isSensorStream,i,2011-06-30T17:00:00.000,,,,Camp Verde T11,,96.8

For XML files, the columns can be in any order as long as the column names correspond to the schema names.

You can omit any fields rather than leaving them empty (null value). For example:

<Positions ESP_OPS="i" BookId="Book1" Symbol="MSFT" SharesHeld="3000"

Basis="528400"/>

Binary files recorded in previous releases cannot be played back until they are converted to the new binary format using streamingconvert. See streamingconvert in the SAP HANA Smart Data Streaming: Utilities

Guide.

SAP HANA smart data streaming records in .rec format, preserving the original timing of the incoming data.

Note

Binary messages are architecture-dependent. Binary messages created in a big-endian machine cannot be loaded into the smart data streaming server running in a little-endian machine, and vice-versa.

In this section:

Recording Incoming Data in a Playback File [page 132]

Record incoming data that is flowing into smart data streaming to a playback file that you can save and view at a later time.

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

131

Playing Recorded Data [page 133]

View and play previously recorded data in a running smart data streaming instance.

Related Information

SAP HANA Smart Data Streaming: Utilities Guide

7.6.1 Recording Incoming Data in a Playback File

Record incoming data that is flowing into smart data streaming to a playback file that you can save and view at a later time.

Prerequisites

Connect to an SAP HANA smart data streaming server. Have your streams and windows visible in the Stream

View.

Procedure

1. Open the SAP HANA Streaming Run-Test perspective.

2. In the Playback view, click Select Project .

3. Select the project you want to record. Click OK.

4. Click the Record icon.

5. Select the streams and windows to record, or click Select All to record all streams and windows in the project. Click OK.

6. Select or create a file in which to save the recording. Click OK.

7. Send data to your selected streams using either:

○ The Manual Input view to input data and publish to your streams, or,

○ File Upload to retrieve an existing data file and publish to your streams.

8. Click Stop to stop recording.

Task overview:

Playback View [page 129]

132

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

Related Information

Playing Recorded Data [page 133]

7.6.2 Playing Recorded Data

View and play previously recorded data in a running smart data streaming instance.

Context

Note

You may select Playback view options before or after you select a file for playback.

Procedure

1. Click Playback File .

2. Browse for and select the file you want to play back.

The playback file is added to the Playback File History. You can also play back a file registered in the history. Double-click a history entry to activate it for playback.

Note

You can delete an item from the history using the either the Remove button or Delete key.

Modifications to the playback history are permanent.

3. Click Play to begin playback.

By default, the data appears in the Stream View at the rate it was recorded.

Task overview:

Playback View [page 129]

Related Information

Recording Incoming Data in a Playback File [page 132]

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

133

7.7 Debugging

Use the debugging tools during project development, not while smart data streaming is in production mode.

Debugging features are normally disabled. Place the system in Trace mode before using the debugging features.

Studio offers an extensive suite of tools for debugging projects, but you can debug from the command line as well. See the SAP HANA Smart Data Streaming: Utilities Guide.

In this section:

Event Tracer View [page 134]

The event tracer is a tool for debugging data flow. It shows the impact an event has on each stream and window of the project.

Debugging with Breakpoints and Watch Variables [page 137]

Use studio to control a running project by enabling tracing, pausing, resuming, and stepping of data flow through smart data streaming streams. You can also create breakpoints and watch variables on a running application.

Related Information

SAP HANA Smart Data Streaming: Utilities Guide

7.7.1 Event Tracer View

The event tracer is a tool for debugging data flow. It shows the impact an event has on each stream and window of the project.

The event tracer view shows the transaction flow through the model and lets you view data in each node

(stream or window). The nodes depicted in the event tracer view are drawn as a data flow, depicting the relationships between the nodes. The following table describes the function of each available button in the event tracer view.

Button

Select Running Project

Layout TopDown

Layout Left to Right

Save

Function

Presents a list of running projects available to monitor from studio.

Arranges shapes vertically for a top-to-bottom data flow.

Arranges shapes horizontally for a left-to-right data flow.

Saves the image as a JPG file.

134

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

Button

Zoom In

Zoom Out

Zoom Page

Print Performance Data to Console

Close Subscription

Show Current Subscription in New

View

Fit Shape Ids

Initialize With Base Data

Function

Enlarges the size of the image.

Reduces the size of the image.

Restores the size of the image to its original size.

Prints the collected data to the console.

Closes the subscription and clears the view.

Displays the current subscription in a separate view.

Expands a shape to show the name of the stream or window.

Sends all event data from smart data streaming through the event tracer.

In this section:

Tracing Data Flow in the Event Tracer [page 135]

Run the event tracer to trace data flow.

Viewing the Topology Stream [page 136]

The Topology Stream constructs the data-flow diagram, where relationships between the nodes of a project are represented as line segments.

Parent topic: Debugging [page 134]

Related Information

Debugging with Breakpoints and Watch Variables [page 137]

7.7.1.1 Tracing Data Flow in the Event Tracer

Run the event tracer to trace data flow.

Prerequisites

The smart data streaming server is running.

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

135

Procedure

1. In studio, in the SAP HANA Streaming Run-Test perspective:

Method Procedure

Event Tracer 1. Click the event tracer view.

2. Click Select Running Project ( ) to show running projects that contain streams or windows.

3. Select a running project for the event tracer.

4. Click OK.

Server View 1. Select the server view.

2. To refresh the server view, click Reconnect All.

3. Select a running project that contains streams.

4. Right-click the project node, and select Show in Event Tracer View .

The nodes depicted in the viewer are drawn as a data flow. As a transaction is processed by each node, the color of the node changes to reflect the type of transaction.

2. Double-click a node to show the corresponding stream's data in the console view.

3. Load test data to view the impact on each stream in the event tracer. Do one of the following:

○ Click the Upload File tab in the toolbar below the Activate Project pane to upload data from a file.

○ In the manual input view, manually enter individual transactions by clicking the Select Stream icon.

Select a stream. To confirm, click OK.

Results

The shapes in the event tracer view change color.

7.7.1.2 Viewing the Topology Stream

The Topology Stream constructs the data-flow diagram, where relationships between the nodes of a project are represented as line segments.

Procedure

1. In the SAP HANA Streaming Run-Test perspective, select Event Tracer view.

2. Click Select Running Project. Select the desired project, and click OK.

3. To view the entire diagram, select Layout top down or Layout left to right.

4. To view a particular node, select the section of the data-flow diagram that contains the desired stream.

136

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

7.7.2 Debugging with Breakpoints and Watch Variables

Breakpoints are locations in a stream or window's input or output that stop the flow of data in the smart data streaming model. A watch variable inspects the data. The following table describes the function of each studio breakpoint button:

Button

Trace On

Trace Off

Step Project

Pause Project

Function

Instructs smart data streaming to begin tracing (debugging). Set this parameter to use the smart data streaming breakpoint APIs.

Stops tracing (debugging).

Steps through the running smart data streaming project.

Pauses playback for projects recorded as .rec files.

Note

When the project is paused, the records from Manual Input and File Upload cannot be updated to streams until the project is resumed.

Enable All Breakpoints

Disable All Breakpoints

Insert Breakpoint

Insert Watch

Print Breakpoint Data to

Console

Enables all breakpoints in the list.

Disables all breakpoints in the list.

Inserts a breakpoint item into the watch table.

Inserts a watch item into the watch table.

Prints the breakpoint and pause state data for the current smart data streaming to the con sole.

Breakpoint Commands

The following breakpoint commands initiate long-running operations. Each of these can be cancelled before completion by clicking Cancel Current Step.

Button

Step Quiesce from Base

Step Quiesce

Step Transaction

Step Quiesce Downstream

Function

Automatically steps all the derived (non-base) streams until their input queues are empty.

Automatically steps the stream and all its direct and indirect descendants until all of them are quiesced.

Automatically steps until the end of transaction.

Steps the descendants of the stream but not the stream itself.

Note

Breakpoints and watch variables persist to the workspace.

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

137

In this section:

You can insert a breakpoint for any stream in the project.

Insert watch variables into the watch table of the Breakpoints view in the Debugger to inspect data as it flows through the project.

Pause SAP HANA smart data streaming while playing back projects with .rec file types.

Parent topic: Debugging [page 134]

Single-step through a project in SAP HANA smart data streaming.

Related Information

Event Tracer View [page 134]

7.7.2.1 Breakpoints

You can insert a breakpoint for any stream in the project.

Breakpoint types include:

Local breaks on input to the stream.

Input breaks on a specific input stream to a stream (only Flex, join, and union can have multiple input streams).

Output breaks when data is output from the stream.

A breakpoint can be associated with a counter (enableEvery). When a counter (n) is associated with a breakpoint, the breakpoint triggers after n events flow through the breakpoint. The counter is then reset to zero.

In this section:

Adding Breakpoints [page 139]

Add breakpoints to streams in smart data streaming.

Parent topic: Debugging with Breakpoints and Watch Variables [page 137]

138

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

Related Information

Adding Watch Variables [page 140]

7.7.2.1.1 Adding Breakpoints

Add breakpoints to streams in smart data streaming.

Prerequisites

● The Debugger view of the SAP HANA Streaming Run-Test perspective in studio is open.

● Trace mode is enabled.

Procedure

1. Click Trace On.

2. Click Insert Breakpoint ( ).

3. Select the stream where you want to set a breakpoint.

4. Select the type of stream.

5. Specify when the breakpoint should trigger by entering a value in the enableEvery field.

6. Click Add.

The selected stream appears in the table within the Insert Breakpoint dialog box.

7. Click OK.

The breakpoint appears in the Debugger view within the Breakpoint table.

8. Right-click the breakpoint and select an option:

○ Enable Breakpoint

○ Disable Breakpoint

○ Remove Breakpoint

9. (Optional) To enable or disable all breakpoints, select either Enable All Breakpoints or Disable All

Breakpoints.

10. (Optional) To remove all breakpoints, right-click within the Breakpoints table and select Remove All

Breakpoints.

11. Click Trace Off to run smart data streaming.

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

139

7.7.2.2 Watch Variables

Insert watch variables into the watch table of the Breakpoints view in the Debugger to inspect data as it flows through the project.

A watch corresponds to:

● Current input of a stream

● Current output of a stream

● Queue of a stream

● Transaction input of a stream

● Transaction output of a stream

● Output history of a stream

● Input history of a stream

Add the watches you want to monitor to the watch table before running smart data streaming. When smart data streaming runs, the watch table is updated dynamically as run-control events (run, step, pause) are sent through smart data streaming.

In this section:

Add a watch element to a breakpoint.

Parent topic: Debugging with Breakpoints and Watch Variables [page 137]

Related Information

Debugging with Breakpoints and Watch Variables [page 137]

7.7.2.2.1 Adding Watch Variables

Add a watch element to a breakpoint.

Prerequisites

● The Debugger view of the SAP HANA Streaming Run-Test perspective in studio is open.

● Trace mode is enabled.

140

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

Procedure

1. Click Trace On.

2. Right-click in the Watch table.

3. Select Add Watch.

4. Select a stream from the Watch Choices box.

5. Select the type of watch you want to set up on that stream.

6. Click Add.

The watch appears in the table at the bottom of the dialog box.

7. Click OK.

The watch appears in the Watch table in the Debugger view.

8. To remove watches, right-click within the Watch table and select, either:

○ Remove Watch to remove a single select watch variable, or,

○ Remove All Watches to remove all watch variables.

7.7.2.3 Pausing SAP HANA Smart Data Streaming

Pause SAP HANA smart data streaming while playing back projects with .rec file types.

Prerequisites

● The Debugger view of the SAP HANA Streaming Run-Test perspective is open.

● Trace mode is enabled.

Procedure

1. In the Debugger, click Pause Project ( ).

2. To resume smart data streaming, click Resume Project, or click Trace Off to close the debugger.

Task overview:

Related Information

Debugging with Breakpoints and Watch Variables [page 137]

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

P U B L I C

141

7.7.2.4 Stepping Through a Project in Studio

Single-step through a project in SAP HANA smart data streaming.

Prerequisites

● The Debugger view of the SAP HANA Streaming Run-Test perspective is open.

● The project is paused.

Procedure

1. In the Debugger view, click Step Project ( ) to perform the next step in the project.

2. Click Cancel Current Step to terminate the action.

Task overview:

Related Information

142

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Running and Testing Streaming Projects

8 CCL Query Construction

Use a CCL query to produce a new derived stream or window from one or more other streams or windows. You can construct a query to filter data, combine two or more queries, join multiple datasources, use pattern matching rules, and aggregate data.

You can use queries only with derived elements, and can attach only one query to a derived element. A CCL query consists of a combination of several clauses that indicate the appropriate information for the derived element. A query is used with the AS clause to specify data for the derived element.

You can also use the generic derived stream and derived window shapes in studio to create complex queries.

Choose the element type according to your input, output, and retention requirements for data, and for preserving insert, update, and delete operations.

The following table specifies the rules for derived streams, derived windows, and derived keyed streams.

Element Input Output

Derived Stream

Derived Window

Another stream

Another stream or window

Stream

Window

Retains state Preserves inserts, up dates, and deletes

No No

As defined in retention pol icy (default is keep all rows)

Yes

Note

To derive a window from a stream, include a

GROUP BY clause in the query.

Derived Keyed

Stream

Another stream or window

Stream As defined in retention pol icy; otherwise no

Yes

In this section:

Splitting Up Incoming Data [page 147]

Accomplish most common querying tasks (filter, aggregate, join, compute, union, and pattern) using a set of queries available in the visual editor, or their equivalents in CCL.

Filtering [page 146]

Use the WHERE clause in your CCL query to filter data to be processed by the derived elements

(streams, windows, or keyed streams).

Use the SPLITTER construct to separate incoming data according to filtering rules and write it out to different target streams.

Unions [page 148]

Use a UNION operator in your CCL query to combine the results of two or more queries into a single result.

Joins [page 150]

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

143

Use joins to combine multiple datasources into a single query.

Pattern Matching [page 159]

Use the MATCHING clause in your CCL query to take input from one or more elements (streams, windows, or keyed streams), and produce records when a prescribed pattern is found within the input data.

Aggregation [page 161]

An aggregation window collects input records and groups them based on the values in the columns specified with the GROUP BY clause, producing one row of output per group. The output record for each group can contain summary statistics about the records in the group, such as average, sum, count, min, max, etc.

Working with Reference Table Queries [page 164]

Reference table queries enable you to look up information in an SAP HANA database table in response to an incoming event.

Creating and Modifying Compute Queries [page 175]

In the visual editor, produce a simple query that transforms the schema or field values of each incoming record. Each incoming event produces one new output event from the fields defined by the column expressions.

Related Information

CCL Continuous Queries [page 55]

8.1 Simple Queries

Accomplish most common querying tasks (filter, aggregate, join, compute, union, and pattern) using a set of queries available in the visual editor, or their equivalents in CCL.

The tools for these six queries are available as objects in the palette, in Streams and Windows:

Filter allows you to filter a stream down to only the events of interest, based on a filter expression.

Similar to the SQL WHERE clause.

Aggregate allows you to group events that have common values and compute summary statistics for the group, such as an average. You can also define a window size, based on either time or number of events. Uses the CCL GROUP BY clause, similar to SQL GROUP BY.

Join allows you to combine records from multiple streams or windows, forming a new record with information from each source. Comparable to a join in SQL, where you specify two or more sources in the FROM clause.

Compute allows you to create a new event with a different schema, and compute the value to be contained in each column (field) of the new event. Comparable to a projection in SQL, where you use a SELECT statement to specify the column expressions, and FROM to specify a single source.

144

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

Union

Pattern allows you to combine multiple streams or windows that all share a common schema into a single stream or window. Similar to the SQL UNION operator.

allows you to watch for patterns of events within a single stream or window or across multiple streams and windows. When smart data streaming detects an event pattern in a running project, it produces an output event. This uses the CCL MATCHING clause.

The following table summarizes the CCL equivalents for simple queries:

Simple Query

Filter

Aggregate

Join

Compute

Union

Pattern

CCL

WHERE clause

GROUP BY clause

FROM clause, WHERE clause, ON clause

Simple SELECT statement, with column expressions

UNION clause

MATCHING clause

Simple Queries from CCL Statements

If you create queries in CCL and want them to appear as simple query shapes in the visual editor, insert a comment immediately preceding the CREATE STREAM or CREATE WINDOW statement, in the following form, where

<QUERY-TYPE>

is the shape name in the visual editor.

/**@SIMPLEQUERY= <QUERY-TYPE> */

For example, this comment causes a CREATE WINDOW statement to map to an aggregate shape in the visual editor: /**@SIMPLEQUERY=AGGREGATE*/ .

Without this comment immediately preceding the CREATE WINDOW statement, the visual editor shows the generic derived window shape.

Note

You cannot modify CCL code in the CCL editor and in the visual editor concurrently. If the visual editor is open, then the CCL editor becomes read-only.

CCL Statements from Simple Queries

When you create a simply query from the palette, the CCL element it creates is based on these rules:

● If the input for the filter, compute, or union object is:

○ A stream, the object creates a stream.

○ A window or Flex stream, the object creates a window.

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

145

● If a join object takes input only from streams, then the join object creates a stream. If the input is from one or more windows or Flex streams, then the join object creates a window. In a stream-window join, the join object creates a stream.

● All aggregate objects create a window.

● All pattern objects create a stream.

Related Information

Connecting Elements [page 112]

Shape Reference [page 68]

Creating and Modifying Aggregate Queries [page 163]

8.2 Filtering

Use the WHERE clause in your CCL query to filter data to be processed by the derived elements (streams, windows, or keyed streams).

The WHERE clause restricts the data captured by the SELECT clause, reducing the number of results generated. Only data matching the value specified in the WHERE clause is sent to your derived elements.

The output of your derived element consists of a subset of records from the input. Each input record is evaluated against the filter expression. If a filter expression evaluates to false (0), the record does not become part of the derived element.

When using the WHERE clause with a keyed stream, filter on columns that have values that do not change between an insert event and subsequent update and delete events. If the columns change, related events can be lost; for example, downstream elements might receive update or delete events without the insert that provided the data being updated or deleted, or fail to receive a delete for a previous insert.

This example creates a new window, IBMTrades, where its rows are any of the result rows from Trades that have the symbol "IBM":

CREATE WINDOW IBMTrades

PRIMARY KEY DEDUCED

AS SELECT * FROM Trades WHERE Symbol = 'IBM';

In this section:

Creating and Modifying Filter Queries [page 147]

Use the visual editor to produce a simple query that only passes on events with specific characteristics. Filter uses a CCL WHERE clause.

146

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

8.2.1 Creating and Modifying Filter Queries

Use the visual editor to produce a simple query that only passes on events with specific characteristics. Filter uses a CCL WHERE clause.

Procedure

1. In the visual editor palette of the SAP HANA Streaming Development perspective, in Streams and

Windows, click Filter ( ).

2. Select a location in the diagram and click to add the shape.

3. Attach the filter object to any stream, window, or Flex operator. Filter objects can have only one input.

4. Edit the value of the filter expression by selecting the value and changing it as necessary. The default value is 1.

Any expression that evaluates to '1' is true, and passes all records through. A value of zero is false.

5. (Optional) Use the toggle option to designate the filter object as LOCAL or OUTPUT. By default, filters are OUTPUT.

8.3 Splitting Up Incoming Data

Use the SPLITTER construct to separate incoming data according to filtering rules and write it out to different target streams.

When you want to separate incoming data into several subsets and process those subsets differently, use the

CREATE SPLITTER construct. This construct operates like the ANSI case statement: it reads the incoming data, applies the specified filtering conditions, and writes out each subset of the data to one or more target streams.

The target streams are implicitly defined by the compiler. The schema for the target streams are derived based on the column_list specification. All the targets are defined as either local or output, depending on the visibility clause defined for the splitter. The default is local.

Note

When the splitter has an output visibility, output adapters can be directly attached to the splitter targets, even though those targets are implicitly defined.

The first condition that evaluates to true (non-zero value) causes the record as projected in the column_list to be inserted into the corresponding target streams. Subsequent conditions are neither considered nor evaluated.

When the source of a splitter is a window:

● Copy the primary keys as-is. The other columns can be changed.

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

147

● If you use the output in a join, specify a retention policy.

● The output is a stateless object and does not receive updates, only base data.

● Projections that contain a non-deterministic expression may produce unpredictable results.

Example

This example creates a schema named TradeSchema and applies it to the input window Trades.

IBM_MSFT_Splitter evaluates and routes data to one of three output windows. Event records with the symbol IBM or MSFT are sent to the IBM_MSFT_Tradeswin window. Event records where the product of trw.Price * trw.Volume is greater than 25,000 are sent to the Large_TradesWin window. All event records that do not meet the conditions placed on the two previous output windows are sent to the

Other_Trades window.

CREATE SCHEMA TradeSchema (

Id long,

Symbol STRING,

Price MONEY(4),

Volume INTEGER,

TradeTime SECONDDATE

) ;

CREATE INPUT WINDOW Trades

SCHEMA TradeSchema

PRIMARY KEY (Id) ;

CREATE SPLITTER IBM_MSFT_Splitter

WHEN trw.Symbol IN ('IBM', 'MSFT') THEN IBM_MSFT_Trades

WHEN trw.Price * trw.Volume > 25000 THEN Large_Trades

ELSE Other_Trades

SELECT trw. * FROM Trades trw ;

CREATE OUTPUT WINDOW IBM_MSFT_TradesWin

PRIMARY KEY DEDUCED

AS SELECT * FROM IBM_MSFT_Trades ;

CREATE OUTPUT WINDOW Large_TradesWin

PRIMARY KEY DEDUCED

AS SELECT * FROM Large_Trades ;

CREATE OUTPUT WINDOW Other_TradesWin

PRIMARY KEY DEDUCED

AS SELECT * FROM Other_Trades ;

8.4 Unions

Use a UNION operator in your CCL query to combine the results of two or more queries into a single result.

If the union is on a window or keyed stream, duplicate rows are eliminated from the result set due to the primary key. If the union is on a stream, duplicates flow through.

The input for a union operator comes from one or more streams or windows. Its output is a set of records representing the union of the inputs. This example shows a simple union between two windows, InStocks and

InOptions:

CREATE INPUT WINDOW InStocks

SCHEMA StocksSchema

148

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

Primary Key (Ts);

CREATE INPUT WINDOW InOptions

SCHEMA OptionsSchema

Primary Key (Ts);

CREATE OUTPUT WINDOW Union1

SCHEMA OptionsSchema

PRIMARY KEY DEDUCED

AS SELECT s.Ts as Ts, s.Symbol as StockSymbol,

Null as OptionSymbol, s.Price as Price, s.Volume as Volume

FROM InStocks s

UNION

SELECT s.Ts as Ts, s.StockSymbol as StockSymbol,

s.OptionSymbol as OptionSymbol, s.Price as Price,

s.Volume as Volume

FROM InOptions s;

In this section:

Creating and Modifying Union Queries [page 149]

Use the visual editor to produce a union query, which combines two or more input streams or windows into a single output. All inputs must have matching schema.

Example: Merging Data from Streams or Windows [page 150]

Use the UNION clause to merge data from two streams or windows and produce a derived element

(stream, window, or keyed stream).

8.4.1 Creating and Modifying Union Queries

Use the visual editor to produce a union query, which combines two or more input streams or windows into a single output. All inputs must have matching schema.

Procedure

1. In the visual editor palette of the SAP HANA Streaming Development perspective, in Streams and

Windows, select Union ( ).

2. Select a location in the diagram and click to add the shape.

3. Attach the union object to two or more inputs, which can be streams, windows, or Flex operators.

Note

To add additional inputs to the union object, you can use the Connector tool in the Palette or the Union icon ( ) in the shape toolbar.

4. (Optional) Use the toggle option to designate the union object as LOCAL or OUTPUT. By default, unions are OUTPUT.

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

149

8.4.2 Example: Merging Data from Streams or Windows

Use the UNION clause to merge data from two streams or windows and produce a derived element (stream, window, or keyed stream).

Procedure

1. Create a new window:

CREATE WINDOW

name

You can also create a new stream or keyed stream.

2. Specify the primary key:

PRIMARY KEY (…)

3. Specify the first derived element in the union:

SELECT * FROM StreamWindow1

4. Add the UNION clause:

UNION

5. Specify the second derived element in the union:

SELECT * FROM StreamWindow2

8.5 Joins

Use joins to combine multiple datasources into a single query.

Streams, windows, and keyed streams can participate in a join. A join can contain any number of windows, but only one stream. Self joins are also supported. For example, you can include the same window more than once in a join, provided each instance has its own alias.

In a stream-window join, the target can be a stream or a window with aggregation. Using a window as a target requires an aggregation, because the stream-window join does not have keys and a window requires a key. The

GROUP BY column in aggregation automatically forms the key for the target window.

Note

Unnamed windows are implicitly created when using a join with a window that produces a stream. The unnamed window is created to ensure that a join does not see records that have not yet arrived at the join.

This can happen because the source to the join and the join itself are running in separate threads.

Joins are performed in pairs, but you can combine multiple joins to produce a complex multitable join.

Depending on the complexity and nature of the join, the compiler may create intermediate joins. The comma

150

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

join syntax supports only inner joins, and the WHERE clause in this syntax is optional. If you omit the WHERE clause, the join creates a many-many relationship between the streams in the FROM clause.

Joins in ANSI syntax can add the DYNAMIC modifier to a window or stream to indicate that its data changes frequently. A secondary index is created on windows joining with an incomplete primary key of a DYNAMIC window or stream. This improves performance but uses additional memory proportional to the total data length of key columns in the index. By default, windows and streams are STATIC and no secondary indexes are created.

The following table describes the join types supported by smart data streaming.

Join Type

Inner Join

Left Outer Join

Right Outer Join

Full Outer Join

Syntax

INNER JOIN

LEFT JOIN

RIGHT JOIN

FULL JOIN

Description

One record from each side of the join is required for the join to pro duce a record.

A record from the left side (outer side) of the join is produced re gardless of whether a record exists on the right side (inner side).

When a record on the right side does not exist, any column from the inner side has a NULL value.

Reverse of left outer join, where the right side is the outer side and the left side is the inner side of the join.

A record is produced whether there is a match on the right side or the left side of the join.

The following table describes the join cardinalities supported by smart data streaming.

Type

One-One

One-Many

Many-Many

Description

Keys of one side of the join are completely mapped to the keys of the other side of the join. One incoming row produces only one row as output.

One record from the one side joins with multiple records on the many side. The one side of the join is the side where all the primary keys are mapped to the other side of the join. Whenever a record comes on the one-side of the join, it produces many rows as the output.

The keys of both side of the join are not completely mapped to the keys of the other side of the join. A row arriving on either side of the join has the potential to produce multiple rows as output.

Note

When a join produces multiple rows, the rows are grouped into a single transaction. If the transaction fails, all of the rows are discarded.

This example joins two windows (InStocks and InOptions) using the FROM clause with ANSI syntax. The result is an output window:

CREATE INPUT Window InStocks SCHEMA StocksSchema Primary Key (Ts) ;

CREATE INPUT Window InOptions SCHEMA OptionsSchema Primary Key (Ts) KEEP ALL ;

CREATE Output Window OutStockOption SCHEMA OutSchema

Primary Key (Ts)

KEEP ALL

SELECT InStocks.Ts Ts,

InStocks.Symbol Symbol,

InStocks.Price StockPrice,

InStocks.Volume StockVolume,

InOptions.StockSymbol StockSymbol,

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

151

InOptions.OptionSymbol OptionSymbol,

InOptions.Price OptionPrice,

InOptions.Volume OptionVolume

FROM InStocks JOIN InOptions

InStocks.Symbol = InOptions.StockSymbol and

InStocks.Ts = InOptions.Ts ;

Keyed Stream Joins

Keyed streams are stateless except when performing a join with a window. In this type of join, the keyed stream uses memory to store a reference to the records in the window.

Restrictions on joins:

● A keyed stream can only be an outer member of an outer join (inner joins are supported).

● A keyed stream cannot participate in a full join.

● When you join a keyed stream with a window, only events that arrive in the keyed stream trigger a join; changes to the window do not.

● A keyed stream cannot be the target of a join when all inputs to the join are windows (named or unnamed).

Nested Joins

Implementing a nested join requires several important functions. Nested join syntax is supported in CCL, but you cannot create or edit a nested join in the visual editor. When a nested join is defined in the CCL file, and you switch to the visual editor, you see an empty join compartment.

In this section:

Using the visual editor, produce a simple query that combines fields from multiple input events into a single output event.

Key field rules ensure that rows are not rejected due to duplicate inserts or the key fields being NULL.

Examples of different join types using the ANSI syntax.

participate in a join. For details of supported joins, see Joins [page 150]

An example of a complex join using the comma-separated syntax.

152

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

8.5.1 Creating and Modifying Join Queries

Using the visual editor, produce a simple query that combines fields from multiple input events into a single output event.

Procedure

1. In the visual editor palette, in Streams and Windows, select Join ( ).

2. Select a location in the diagram and click to add the shape.

3. Connect the join object to the streams or windows that provide the inputs to the join.

Connect join objects to two or more streams, windows, or Flex operators. Join objects can take input from two or more objects, but can produce only one output.

Note

Streams, windows and keyed streams can participate in a join. However, only one stream can

Tip

To add multiple connections, Shift+click while holding the Connector tool. To return to normal selection, press Esc or click the Select tool in the palette to release it.

4. Use Copy Columns from Input to select input fields to include in the output of this query.

5. Add column expressions , as necessary.

6. Edit a column expression by selecting one of the tabs in the Properties view and editing the corresponding fields, or by:

○ Double-clicking to open the inline editor.

○ Selecting an expression and pressing Ctrl+F2 to edit it using the pop-up expression editor.

Tip

When entering column names and their datatypes, use Tab to easily move between cells in the table.

7. Click Add Join Condition to specify the columns to use to match incoming events across the different sources.

Complete the Edit Join Expression dialog to define the join type, data sources for the ON clause, and any other join constraints.

Note

If you create a join using comma-separated syntax in the CCL editor, and then add an ON clause using the Edit Join Expression dialog in the visual editor, the WHERE clause initially created in the commaseparated syntax will not be removed. This does not affect the result, however it will negatively affect performance.

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

153

If you do not see the columns you want in the Edit Join Expression dialog, ensure you have connected the join object to the correct input sources.

8. To join a column to itself, click Add Input Alias in the shape toolbar.

A column alias is required to provide a unique name for each join condition.

9. (Optional) Use the toggle option to designate the join object as LOCAL or OUTPUT. By default, joins are

OUTPUT.

10. (Optional) Select Set Keep Policy and choose an option.

To edit the keep policy, right-click the input window or stream in the Inputs menu. Select Set Keep Policy to add a keep policy, and Delete Keep Policy to remove it.

Task overview:

Joins [page 150]

Related Information

Parent topic: Joins [page 150]

8.5.2 Key Field Rules

Key field rules ensure that rows are not rejected due to duplicate inserts or the key fields being NULL.

● The key fields of the target are always derived completely from the keys of the 'many' side of the join. In a many-many relationship, the keys are derived from the keys of both sides of the join.

● In a one-one relationship, the keys are derived completely from either side of the relationship.

● In an outer join, the key fields are derived from the outer side of the join. An error is generated if the outer side of the join is not the many-side of a relationship.

● In a full-outer join, the number of key columns and the type of key columns need to be identical in all sources and targets. Also, the key columns require a firstnonnull expression that includes the corresponding key columns in the sources.

When the result of a join is a window, specific rules determine the columns that form the primary key of the target window. In a multitable join, the same rules apply because conceptually each join is produced in pairs, and the result of a join is then joined with another stream or window, and so on.

The following table summarizes the key field rules for joins:

154

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

Join Type

INNER

LEFT

RIGHT

OUTER

One-One

Include keys from at least one side in the projection list

(or a combina tion of them if keys are com posite).

Include keys from the left side alone.

Include keys from the right side alone.

Form keys us ing firstnonnu ll

() on each pair of keys from both sides.

One-Many

Include keys from the right side in the projec tion list.

Not allowed.

Include keys from the right side in the projec tion list.

Not allowed.

Many-One Many-Many

Include keys from the left side in the projection list.

Include keys from both sides in the projection list.

Include keys from the left side in the projection list.

Not allowed.

Related Information

Parent topic: Joins [page 150]

8.5.3 Join Examples: ANSI Syntax

Examples of different join types using the ANSI syntax.

Refer to these inputs for the examples below:

CREATE INPUT STREAM S1 SCHEMA (Val1S1 integer, Val2S1 integer, Val3S1 string);

CREATE INPUT WINDOW W1 SCHEMA (Key1W1 integer, Key2W1 string, Val1W1 integer,

Val2W1 string) PRIMARY KEY (Key1W1, Key2W1);

CREATE INPUT WINDOW W2 SCHEMA (Key1W2 integer, Key2W2 string, Val1W2 integer,

Val2W2 string) PRIMARY KEY (Key1W2, Key2W2);

CREATE INPUT WINDOW W3 SCHEMA (Key1W3 integer, Val1W3 integer, Val2W3 string)

PRIMARY KEY (Key1W3);

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

155

Simple Inner Join: One-One

Here, keys can be derived from either W1 or W2:

CREATE OUTPUT WINDOW OW1

PRIMARY KEY (Key1W2, Key2W2)

AS SELECT W1.*, W2.*

FROM W1 INNER JOIN W2 ON W1.Key1W1 = W2.Key1W2 AND W1.Key2W1 = W2.Key2W2;

Simple Left Join: One-One

The keys are derived from the outer side of the left join. It is incorrect to derive the keys from the inner side because the values could be null:

CREATE OUTPUT WINDOW OW2

PRIMARY KEY (Key1W1, Key2W1)

AS SELECT W1.*, W2.*

FROM W1 LEFT JOIN W2 ON W1.Key1W1 = W2.Key1W2 AND W1.Key2W1 = W2.Key2W2;

Simple Full Outer Join: One-One

The key columns all have a required firstnonnull expression in it:

CREATE OUTPUT WINDOW OW3

PRIMARY KEY (Key1, Key2)

AS SELECT firstnonnull(W1.Key1W1, W2.Key1W2) Key1, firstnonnull(W1.Key2W1,

W2.Key2W2) Key2, W1.*, W2.*

FROM W1 FULL JOIN W2 ON W1.Key1W1 = W2.Key1W2 AND W1.Key2W1 = W2.Key2W2;

Simple Left Join: Many-One

All the keys of W2 are mapped and only one key of W1 is mapped in this join. The many-side is W1 and the oneside is W2. The keys are derived from the many-side:

CREATE OUTPUT WINDOW OW4

PRIMARY KEY (Key1W1, Key2W1)

AS SELECT W1.*, W2.*

FROM W1 LEFT JOIN W2 ON W1.Key1W1 = W2.Key1W2 AND W1.Val2W1 = W2.Key2W2;

156

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

Simple Left Join: Many-One (DYNAMIC Modifier)

W3 is DYNAMIC and only one key of W1 is mapped in this join, so a secondary index is created on W1. W1 is also

DYNAMIC, but all keys of W3 are mapped, so no secondary index is created on W3:

CREATE OUTPUT WINDOW OW5

PRIMARY KEY DEDUCED

AS SELECT W1.*, W3.*

FROM W1 (DYNAMIC) LEFT JOIN W3 (DYNAMIC) ON W1.Key1W1 = W3.Key1W3;

Simple Inner Join: Many-Many

This is a many-many join because neither of the keys is fully mapped. The keys of the target are the keys of all the windows participating in the join:

CREATE OUTPUT WINDOW OW6

PRIMARY KEY (Key1W1, Key2W1, Key1W2, Key2W2)

AS SELECT W1.*, W2.*

FROM W1 JOIN W2 ON W1.Val1W1 = W2.Val1W2 AND W1.Val2W1 = W2.Val2W2;

Simple Stream-Window Left Join

When a left join involves a stream, the stream is on the outer side. The target cannot be a window unless it is also performing aggregation:

CREATE OUTPUT STREAM OSW1

AS SELECT S1.*, W2.*

FROM S1 LEFT JOIN W2 ON S1.Val1S1 = W2.Key1W2 AND S1.Val3S1 = W2.Key2W2;

Complex Window-Window Join

The keys for OW4 can be derived either from W1 or W2 because of the inner join between the two tables:

CREATE OUTPUT WINDOW OW7

PRIMARY KEY DEDUCED

AS SELECT S1.*, W1.*, W2.*, W3.* //Some column expression.

FROM S1 LEFT JOIN (W1 INNER JOIN (W2 LEFT JOIN W3 ON W2.Key1W2 = W3.Key1W3) ON

W1.Key1W1 = W2.Key1W2 AND W1.Key2W1 = W2.Key2W2) ON S1.Val1S1 = W1.Key1W1

WHERE W2.Key2W2 = 'abcd'

GROUP BY W1.Key1W1, W2.Key2W2

HAVING SUM(W3.Val1W3) > 10;

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

157

Complex Stream-Window Join

Here, the join is triggered only when a record arrives on S1. Also, because there is aggregation, the target is a window instead of being restricted to a stream:

CREATE OUTPUT WINDOW OW8

PRIMARY KEY DEDUCED

AS SELECT S1.*, W1.*, W2.*, W3.* //Some column expression.

FROM S1 LEFT JOIN (W1 INNER JOIN (W2 LEFT JOIN W3 ON W2.Key1W2 = W3.Key1W3) ON

W1.Key1W1 = W2.Key1W2 AND W1.Key2W1 = W2.Key2W2) ON S1.Val1S1 = W1.Key1W1

WHERE W2.Key2W2 = 'abcd'

GROUP BY W1.Key1W1, W2.Key2W2

HAVING SUM(W3.Val1W3) > 10;

Related Information

Parent topic: Joins [page 150]

8.5.4 Join Example: Comma-Separated Syntax

An example of a complex join using the comma-separated syntax.

This join is a complex join of three windows using the comma-separated join syntax. The WHERE clause specifies the conditions on which records are joined.

CREATE OUTPUT WINDOW OW4

PRIMARY KEY DEDUCED

AS SELECT W1.*, W2.*, W3.*

FROM W1, W2, W3

WHERE W1.Key1W1 = W2.Key1W2 AND W1.Key2W1 = W2.Key2W2 AND W1.Key1W1 = W3.Key1W3;

Related Information

Creating and Modifying Pattern Queries [page 160]

158

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

8.6 Pattern Matching

Use the MATCHING clause in your CCL query to take input from one or more elements (streams, windows, or keyed streams), and produce records when a prescribed pattern is found within the input data.

Pattern streams can check whether events (rows from the input sources matching certain criteria) occur during a specific time interval, then send records to downstream streams if a match has occurred.

Pattern matching can be used to distill complex relationships between data into compact and easilymaintainable expressions.

Note

The pattern rule engine uses any incoming event in order to match the defined pattern, regardless of the opcode of the incoming event. Include the opcode in each event's definition to filter out unwanted rows.

This example creates an output stream, ThreeTrades, which monitors the QTrades streams and sends a new event when it detects three trades on the same symbol within five seconds. The output of this stream is the symbol of the traded stock, and its latest three prices. The trades do not have to occur consecutively, but must occur within five seconds of each other. Multiple patterns may be in the process of being matched at the same time.

CREATE OUTPUT STREAM ThreeTrades

SELECT

T1.Symbol,

T1.Price Price1,

T2.Price Price2,

T3.Price Price3

FROM

QTrades T1,

QTrades T2,

QTrades T3

MATCHING[5 SECONDS: T1, T2, T3]

ON T1.Symbol = T2.Symbol = T3.Symbol;

For details on the MATCHING clause, see the SAP HANA Smart Data Streaming: CCL Reference Guide.

In this section:

In the visual editor, run a pattern-matching query that watches for a specific pattern of incoming events on one or more inputs and produces an output event when the pattern is detected. Pattern uses the CCL MATCHING clause.

Related Information

SAP HANA Smart Data Streaming: CCL Reference

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

159

8.6.1 Creating and Modifying Pattern Queries

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor palette, in Streams and Windows, click Pattern ( ).

3. Select a location in the diagram and click to add the shape.

4. Connect the Pattern shape to one or more streams or windows that are the inputs to query.

5. Add columns: a. Click Copy Columns from Input ( ) in the shape toolbar to select the columns to copy into the schema for the Pattern query.

This is the schema of the new event that is produced when the pattern is detected.

b. Add additional columns by clicking Add Column Expression in the shape toolbar.

6. Edit a column expression by selecting one of the tabs in the Properties view and editing the corresponding fields, or by:

○ Double-clicking to open the inline editor.

○ Selecting an expression and pressing Ctrl+F2 to edit it using the pop-up expression editor.

Tip

When entering column names and their datatypes, use Tab to easily move between cells in the table.

7. Create and edit a pattern expression: a. Click Add Pattern ( ).

b. Enter an alias for the event.

c. Enter either a time interval or parameters.

d. To define the expression, right-click Pattern to add an event. Continue right-clicking elements of the expression to add operators and refine the event expression. Then click Next.

e. Click Add to add a join condition.

8. (Optional) Use the toggle option to designate the pattern object as LOCAL or OUTPUT. By default, patterns are OUTPUT.

160

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

8.7 Aggregation

All records in each group have the same values for the columns specified in the GROUP BY clause. All columns specified in the GROUP BY clause must also be included in the SELECT clause, because these columns form the primary key for the aggregation window. This is why the primary key for the aggregate window uses the

PRIMARY KEY DEDUCED clause instead of explicitly specifying a primary key.

When using aggregation, consider the memory usage implications. All of the input records for which an aggregate function is to be calculated are stored in memory. The data structure that holds all the records in memory is called the aggregation index.

Using the KEEP Clause for Aggregation Windows

When creating a window in CCL, the KEEP policy can be specified as part of the FROM clause to define an unnamed window, or it can be defined for the window itself. In the case of an aggregation window, the effects of the two are very different and are often misunderstood.

The first (and most common) method is to apply a KEEP clause to the input, creating an unnamed window that will be used for the input to the aggregation. In the following example, the aggregation is applied to an unnamed window that contains all events received from stream Stream123 in the last 5 minutes:

CREATE INPUT STREAM Stream123

SCHEMA (SourceID string ,

Value Integer ,

Tstamp msdate );

/**@SIMPLEQUERY=AGGREGATE*/

CREATE OUTPUT WINDOW MovingAverage

PRIMARY KEY DEDUCED

AS SELECT

Stream1.SourceID SourceID ,

avg ( Stream123.Value ) Value ,

max ( Stream123.Tstamp ) Tstamp

FROM Stream123 KEEP 5 MINUTES

GROUP BY Stream123.SourceID ;

The second method is to apply a KEEP clause to the window being created. Here, the KEEP policy will be applied to the output; it will be applied after aggregation and will be applied to the group records. Thus the

KEEP policy will delete the summary record of any group that hasn't been added or updated recently, according to the KEEP policy. The following example uses this method:

CREATE INPUT WINDOW Window1

SCHEMA (SourceID string ,

Value Integer ,

Tstamp msdate )

PRIMARY KEY (SourceID)

KEEP 1 HOUR;

/**@SIMPLEQUERY=AGGREGATE*/

CREATE OUTPUT WINDOW MovingAverage

PRIMARY KEY DEDUCED

KEEP 5 MINUTES

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

161

AS SELECT

Window1.SourceID SourceID ,

avg ( Window1.Value ) Value ,

max ( Window1.Tstamp ) Tstamp

FROM Window1

GROUP BY Window1.SourceID ;

Note that this example does not create a 5 minute moving average. The KEEP 5 MINUTES policy is applied after aggregation and is applied to the rows in the window MovingAverage. So the average and max values will be computed across all records held in Window1 (which has all record received in the last 1 hour), but any group that has not been added or updated in the last 5 minutes will be deleted from the MovingAverage window. Thus the window MovingAverage has a one hour moving average for all groups added or updated in the last 5 minutes.

Aggregating Over Streams: Avoiding Unbounded Growth

You often need to create an aggregation window that is fed by a stream, allowing you to compute aggregate values over events received from the stream. To compute aggregation values, an aggregation index is created on the input to an aggregation window. When a stream serves as the input to an aggregation window, it’s important to specify a KEEP policy as part of the FROM clause, to limit the size of the aggregation index.

Failure to specify a KEEP policy in the FROM clause will lead to unbounded growth of the aggregation index, eventually consuming all system resources and causing the project to shut down.

To prevent this, construct a SELECT clause that only uses additive aggregation functions. These are special aggregate functions that do not require an aggregation index and are computed incrementally.

When the projection contains only additive aggregation functions, the server performs additive optimization; therefore, there is no need to maintain an aggregation index, and memory does not grow. Supported additive

aggregation functions are sum, count, avg, and valueInserted. For more information, see Improving

Aggregation Performance [page 365]

. Note that last() is not an additive function; you must use valueinserted() rather than last(). Here is an example:

CREATE INPUT STREAM Stream2

SCHEMA (SourceID string ,

Value Integer ,

Tstamp msdate );

/**@SIMPLEQUERY=AGGREGATE*/

CREATE OUTPUT WINDOW MovingAverage

PRIMARY KEY DEDUCED

AS SELECT

Stream2.SourceID SourceID ,

avg ( Stream2.Value ) Value ,

valueInserted ( Stream2.Tstamp ) Tstamp

FROM Stream2

GROUP BY Stream2.SourceID ;

Note

If even one of the non-GROUP BY column expressions does not have an additive aggregation function, the server needs to maintain an aggregation index resulting in unbounded memory growth. If you select a column without specifying an additive aggregation function, it is considered as a non-additive function forcing the server to maintain an aggregation index.

162

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

Advanced Aggregation Techniques

In addition to the GROUP BY clause, you can optionally specify a GROUP FILTER and GROUP ORDER BY clause. The GROUP ORDER BY clause orders the records in a group by the specified columns before applying the GROUP FILTER clause and the aggregation functions. With the records ordered, aggregation functions sensitive to the order of the records such as first, last, and nth can be used meaningfully.

The GROUP FILTER clause is executed after the GROUP ORDER BY clause and eliminates any rows in the group that do not meet the filter condition. The specified filter condition can be used independently of the

GROUP ORDER BY clause, but by ordering the group before filtering, the rank function can be used to filter the group. Here is an example:

After you execute the GROUP ORDER BY clause, every row in the group is ranked from 1 to N. This allows you to specify rank () < 11 in the GROUP FILTER clause, which means that the aggregation function is only applied to the first 10 rows in the group after it has been ordered by the columns specified in the GROUP

ORDER BY clause.

You can also specify an optional HAVING clause, which applies a filter after applying the aggregation functions to the records in a group. In other words, a filter specified in a HAVING clause will filter out the aggregate record for any group that does not meet the filter criteria.

Note

Note: You can only specify GROUP ORDER BY, GROUP FILTER, and HAVING clauses in conjunction with a

GROUP BY clause.

In this section:

Creating and Modifying Aggregate Queries [page 163]

Using the visual editor, produce a simple query that combines data, similar to the CCL GROUP BY,

GROUP FILTER, and GROUP ORDER clauses.

8.7.1 Creating and Modifying Aggregate Queries

Using the visual editor, produce a simple query that combines data, similar to the CCL GROUP BY, GROUP

FILTER, and GROUP ORDER clauses.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor palette, in Streams and Windows, select Aggregate ( ).

3. Select a location in the diagram and click to add the shape.

4. Connect the Aggregate shape to an input.

The aggregate border changes from red to black, indicating that it is valid, now that it has input.

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

163

5. Add columns: a. Click Copy Columns from Input ( ) in the shape toolbar to select the columns to copy into the schema for the Aggregate window.

b. Add additional columns by clicking Add Column Expression in the shape toolbar.

6. Edit a column expression by selecting one of the tabs in the Properties view and editing the corresponding fields, or by:

○ Double-clicking to open the inline editor.

○ Selecting an expression and pressing Ctrl+F2 to edit it using the pop-up expression editor.

Tip

When entering column names and their datatypes, use Tab to easily move between cells in the table.

7. Click Add GroupBy Clause ( ) in the shape toolbar to edit the grouping of columns in the aggregate object.

Note

The Aggregate shape must have exactly one GROUP BY expression.

8. (Optional) Click Set Keep Policy ( ) to create a retention window.

The default policy is to keep all rows of incoming data. You can also choose to keep only the last row, a specific number of rows, or keep the rows for a specific time. This defines the KEEP clause. You can also go further, and retain the rows defined by the KEEP clause to span retention across multiple specified columns. This spanning of retention across columns is done by listing column names in the PER clause.

9. (Optional) Use the Toggle option to designate the aggregate object as LOCAL or OUTPUT. By default, aggregates are OUTPUT.

Related Information

Creating a Reference Table Query Using Schema Discovery [page 167]

8.8 Working with Reference Table Queries

Reference table queries enable you to look up information in an SAP HANA database table in response to an incoming event.

Incoming events can arrive with only a portion of the information necessary to complete the processing you wish to specify in your project. When that additional information is present in existing tables in an SAP HANA database, you can use reference table queries to look it up. There are two parts to this: creating the table reference query, and using it to execute an ad hoc query in a join or Flex operator.

164

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

When creating the reference, determine what data you want to use. Then identify the table containing the data by name, obtain the schema of the table, and find out what service to use to contact the database. Decide whether you want to attempt to reconnect if the connection is dropped, and if so, how many times, and how long to wait between attempts.

When joining a stream or window to a reference, determine what you want as the output of the join. There are numerous restrictions on how to use references in joins, and what types of output you can obtain. For example, if you want the output of the join to be a window, you have to specify the primary key of the reference and use the complete primary key in the ON or WHERE clause of the join.

There are several different ways to use references within CCLScript programs. You can iterate over the rows in the table or grab specific rows. Basically, you can use a reference in the same ways you use a window. It is simply another source of data for processing in your CCLScript routine.

You can use references—in joins and in CCLScript programs—inside a module, as well as within the main body of your project. Like stores, references used in a module must be bound to a reference defined in the main body of your project.

Prerequisites

Install the SAP HANA ODBC client on the system where you want to run projects that include reference table queries. The minimum required version of the SAP HANA ODBC client is 1.0.73, but you should install the most recent version.

Database Service Definition

All connections to external databases, including reference table queries, are made using data services defined in the cluster. You can define or modify a data service definition using the Data Services view in the SAP HANA

Streaming Development perspective. You can also use studio to define the default SAP HANA service entry.

Note

If the SAP HANA database on which you are making these queries is on an SAP HANA cluster, see

Configuring to Support SAP HANA Failover in the SAP HANA Smart Data Streaming: Configuration and

Administration Guide for details on how to add the other nodes in the SAP HANA cluster to the database service definition.

Caching

When a project joins streaming data to SAP HANA tables, such as customer or instrument information, reference table queries may repeatedly make the same requests. Since turning on caching for a reference table query enables local storage of previous query results, caching can improve the performance of the join or

Flex operation using the reference table query. It can also reduce network traffic when the SAP HANA table being queried is on the network.

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

165

By default, caching is turned off (you can explicitly turn it off by setting cachePolicy to

NONE

). Turn it on by setting cachePolicy to

ONACCESS

By default, there are no limits on the size of the cache or the age of the cached query results, so the size of the cache keeps increasing as query results are cached. And, once the query results have been retrieved and cached, they will never be retrieved again.

If you want to be able to retrieve query results, you can specify the maximum physical size of the cache

(maxCacheSizeMB), or how old a cached query result can be (maxCacheAge) before it is no longer returned when the query runs again, or both.

Error Handling

When you start a project that contains a reference table query, the query does a table schema compatibility check. The reference scans the database table and verifies that:

● For each column specified in the reference, there is a column of the same name (case-insensitive) in the table.

● The datatype of the column in the table is compatible with the datatype of the column in the reference.

● If the reference definition specifies a primary key, there is a matching primary key in the database. If the reference definition doesn't specify a primary key, it doesn't matter whether or not the database has a primary key.

To check the type for each mapped column and to be as database-agnostic as possible, the reference attempts to pull a sample row from the database. If it can pull that column into smart data streaming, the check succeeds. Otherwise it fails, except in these cases:

● If the query that the reference uses to do the type-checking is rejected by the database (because it doesn't support SQL 2003 standards), the reference doesn't complete type-checking, but allows the project to start up, providing a warning that it can't guarantee that the type mapping is valid.

● If the table has no data in it, the type-checking stops, and a warning is printed that it can't guarantee that the type mapping is valid.

While a project is running, the error scenarios are mostly connection-based. When a failure is caused by a lost connection, the server attempts to reconnect based on the reconnect parameters specified in the reference's definition.

In this section:

Create a reference in a smart data streaming project that queries a table in an SAP HANA database.

Using a Reference Table Query in a Join [page 169]

This example shows the procedure for creating a reference table query, an input stream, and an output stream that implements a join of data from the reference table query and the input stream to add customer data to an incoming order.

Using a Reference Table Query in a Module [page 170]

This example obtains a bibliography for the specified author using a reference table query inside of a module.

Using a Reference Table Query in a Flex Operator [page 171]

166

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

Use a reference table query to obtain data from an SAP HANA database table for processing by a

CCLScript routine in a Flex operator.

Using a Reference Table Query in CCLScript [page 172]

This example uses a reference table query within a CCLScript routine that iterates over the rows in a table to obtain data about a specified baseball player's batting history that is then used to compute statistics.

Using a Reference Table Query in CCLScript to Get a Single Row from a Table [page 174]

This example uses a reference table query within a CCLScript routine to obtain a single row from a table.

Related Information

SAP HANA Smart Data Streaming: Configuration and Administration Guide

8.8.1 Creating a Reference Table Query Using Schema

Discovery

Create a reference in a smart data streaming project that queries a table in an SAP HANA database.

Context

You can create reference table queries either in the studio visual editor, or directly in CCL. Create a reference table query in CCL using the CREATE REFERENCE statement.

If you create the reference table query using the visual editor, you can:

● Use the schema discovery wizard to fill out the query properties.

● Manually enter values for each property.

● Use a mix of both discovery and manual input through the properties view.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the palette, expand Streams and Windows and select Reference (

3. Select a location in the diagram and click to add the shape.

4. Provide a name for the Reference ( ).

5. Click on Schema Discovery ( ).

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

167

Studio displays the Schema Discovery dialog box. The service field is populated with the name of the default SAP HANA service entry, as defined in studio. If you have not designated a default SAP HANA service entry, you are prompted to do so.

6. In the service field:

○ Leave the specified service name, to use it when connecting to the external database.

○ Replace the specified service with another service, as defined in the studio Data Services View.

○ Click on Discover and select from the list of services displayed.

7. In the Source Schema field:

○ Enter the name of the schema for the external database containing the table the reference will query.

○ Click on Discover and select from the list of schemas displayed.

8. In the Source field:

○ Enter the name of the table the reference will query.

○ Click on Discover and select from the list of tables displayed.

9. Select the Discover Primary Keys checkbox to define primary keys in the reference matching those in the external database table. This is optional, but highly recommended to provide maximum flexibility when using the reference.

10. Click OK.

By default, smart data streaming builds the reference schema based on the schema of the external database table. Once the reference schema has been built, you can remove a column by right-clicking anywhere on the column and choosing delete element. Edit the column properties within the reference shape by double-clicking on the property names or edit them in the Properties view. The datatypes in the source schema for the reference must be compatible with those in the target SAP HANA database table.

You can create a reference within a module; a reference created outside of a module cannot be used in a module. See

Creating a Module in the Visual Editor [page 190]

Results

This method omits several additional properties that you can set through the properties view of the SAP HANA

Streaming Development perspective. See CREATE REFERENCE Statement in the SAP HANA Smart Data

Streaming: CCL Reference for a list of all the properties and their definitions.

Related Information

SAP HANA Smart Data Streaming: CCL Reference

168

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

8.8.2 Using a Reference Table Query in a Join

Procedure

1. Obtain the necessary information about the external database table containing the information you wish to look up:

○ The name of the table containing the data

○ The schema for that table

○ The service to use to connect to it

2. Create the reference table query in your project:

CREATE REFERENCE customerRef

SCHEMA (customerID integer, fullName string, address string)

PRIMARY KEY (customerID)

PROPERTIES service='databaseLookup',source='customerAddr',sourceSchema='addressSchema';

The schema of the reference does not have to include all of the columns in the source table or list them in the same order. For each column that you do specify in the schema, however, the column name must match the column name in the database, and the datatype must be compatible with the datatype in the database.

The customerID column is the primary key in the source table, and it contains a unique number assigned to each customer. A primary key is not required to produce a stream; only a window. If you declare a primary key for the reference, however, it must match the primary key in the table. So, whenever the table has a primary key, it is a good practice to make it the primary key of the reference table query (giving you the flexibility to produce either a stream or a window).

When you specify a primary key in a reference table query, also include the sourceSchema property, which identifies the schema, in this case addressSchema, of the database containing the table.

3. Create the input stream for events to enter the project:

CREATE INPUT STREAM orderStream

SCHEMA (orderID integer, customerID integer, itemID integer);

4. Create an output stream that implements a join of the incoming event with the data obtained from the reference table query:

CREATE OUTPUT STREAM orderWithCustomerInfoStream

SCHEMA (orderID integer, customerName string, customerAddress string)

AS SELECT orderStream.orderID, customerRef.fullName, customerRef.address

FROM orderStream, customerRef

WHERE orderStream.customerID=customerRef.customerID;

While the column names in the output stream's schema declaration, customerName and customerAddress, do not have to match the column names in the table, fullName and address, the datatypes in the output stream's schema must match the datatypes in the table.

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

169

8.8.3 Using a Reference Table Query in a Module

This example obtains a bibliography for the specified author using a reference table query inside of a module.

Context

In this example, an order for a book produces a list of other books by that author. Because this list may be desired in response to other events, it is produced within a module.

Procedure

1. Obtain the necessary information about the external database table containing the information you wish to look up:

○ The name of the table containing the data

○ The schema for that table

○ The service to use to connect to it

2. Create the reference table query in your project. When you define a reference in a module, it is always mapped to a reference in your project that contains the connection information for the database:

CREATE REFERENCE getBiblio

SCHEMA (authorName string, bookTitle string, publisher string)

PROPERTIES service='databaseLookup',source='bibliographies',sourceSchema='bookSchema';

If you think that the reference table query will make the same requests many times, specify a cache. This example shows a cache that can use up to two megabytes of storage and holds a query result for up to two hours:

CREATE REFERENCE getBiblio

SCHEMA (authorName string, bookTitle string, publisher string)

PROPERTIES service='databaseLookup',source='bibliographies',sourceSchema='bookSchema', cachePolicy='ONACCESS',maxCacheSizeMB=2,maxCacheAge=2 hours;

Note

To decide whether to enable caching, consider how often you expect the same request to be made. To decide whether to limit the size of the cache or the amount of time a query result is held, consider the amount and type of data you expect the query to return.

3. Create an input stream for order events to enter the project:

CREATE INPUT STREAM orderStream

SCHEMA (orderID integer, customerID integer, itemID integer, authorName string);

170

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

4. Create a module that uses this table reference query. In the module, you only need to specify the name and schema of the reference table query:

CREATE MODULE listOtherBooks

IN orderStream

OUT otherBooks

REFERENCES getBiblio

BEGIN

CREATE REFERENCE getBiblio

SCHEMA (authorName string, bookTitle string, publisher string);

CREATE INPUT STREAM orderStream

SCHEMA (orderID integer, customerID integer, itemID integer, authorName string);

CREATE OUTPUT STREAM otherBooks

SCHEMA (authorName string, bookTitle string, publisher string)

AS SELECT orderStream.authorName, getBiblio.bookTitle, getBiblio.publisher

FROM orderStream, getBiblio

WHERE orderStream.authorName=getBiblio.authorName;

END;

5. Load the module to generate the list of other works by the author of the book that was ordered. Loading the module creates a mapping between the streams in the main project and in the module. The output from the module goes to the otherBooks local stream in the main project:

LOAD MODULE listOtherBooks AS listAll

orderStream = orderStream

OUT

otherBooks = otherBooks

REFERENCES

getBiblio = getBiblio;

6. Create an output stream to read the local stream otherBooks and display the list of other books by the author:

CREATE OUTPUT STREAM otherBooksOutput

SCHEMA (authorName string, bookTitle string, publisher string)

AS SELECT * FROM otherBooks;

8.8.4 Using a Reference Table Query in a Flex Operator

Use a reference table query to obtain data from an SAP HANA database table for processing by a CCLScript routine in a Flex operator.

Procedure

1. Obtain the necessary information about the external database table containing the information you wish to look up:

○ The name of the table containing the data

○ The schema for that table

○ The service to use to connect to it

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

171

2. Create the reference table query in your project.

3. Create the input stream or window.

4. Create the Flex operator: a. From the palette, under Streams and Windows, select Flex ( ), then click an empty area in the diagram.

b. From the palette, select Connector and connect the input stream or window to the Flex operator.

c. From the palette, select Connector and connect the reference table query to the Flex operator.

d. Define the schema for the Flex operator.

e. (Optional) Click on Aging Policy ( ).

f. (Optional) Click Set Output Keep Policy ( ) to set keep policy options.

5. Implement a CCLScript routine: a. In the Flex operator, under Methods, click on the ( ) of the expression you wish to edit.

b. In the Edit Expression Value dialog box, write the CCLScript script you wish to implement.

See

Using a Reference Table Query in CCLScript [page 172]

and

Using a Reference Table Query in

CCLScript to Get a Single Row from a Table [page 174] for examples of CCLScript routines in a Flex

operator.

8.8.5 Using a Reference Table Query in CCLScript

Procedure

1. Obtain the necessary information about the external database table containing the information you wish to look up:

○ The name of the table containing the data.

○ The schema for that table.

○ The service to use to connect to it.

2. Create the reference table query in your project.

For example, create a reference table query named AtBats:

CREATE REFERENCE AtBats

SCHEMA (teamAtBat INTEGER, player STRING, bases INTEGER)

PRIMARY KEY (teamAtBat)

PROPERTIES service='HANA',source='Batting',sourceSchema='BattingSchema';

The teamAtBat is incremented each time a player bats a ball, providing a unique number for each at-bat by a member of the team, so that column is the primary key in the source table. It must also be the primary key of the reference table query. Because we have specified a primary key in this reference, we have to include the sourceSchema property, which identifies the schema, BattingSchema, of the database containing the table.

172

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

3. Create the input stream for events to enter the project.

For any event that is the appearance, or mention, of a player's name, create an input stream named

StatRequest, consisting of a single string, player:

CREATE INPUT STREAM StatRequest SCHEMA (player STRING);

4. Initialize the Flex operator, named Percentage, to read in the reference query AtBats and the input stream StatRequest, specify the output stream PercentageOutput, and set the variables

<atBats> and

<hits>

to zero:

CREATE FLEX Percentage IN AtBats, StatRequest

OUT OUTPUT STREAM PercentageOutput SCHEMA (atBatPercentage float)

BEGIN

ON StatRequest {

integer atBats := 0;

integer hits := 0;

5. Close the Flex operator by dividing the number of

<hits>

by the number of

to calculate the batting average,

, placing that value in the

column of the output stream, and outputting it:

float percentage := hits/atBats;

output [atBatPercentage=percentage];

};

END;

6. Between the initialization and close of the Flex operator, add a CCLScript routine to iterate through the entries in the Batting table to find out how many times the player was at bat and how many times he hit a ball. There are three ways to do this: a. Use the for and if commands to handle iterating through the records, and take advantage of the

<AtBats_stream>

that is automatically created with the AtBats reference query:

for ( record in AtBats_stream ) {

if ( record.player = StatRequest.player ) {

if ( record.bases > 0 ) {

hits := hits + 1;

}

atBats := atBats + 1;

}

This method has the advantage of being easy to read and maintain.

b. Use the modified for command syntax that incorporates testing whether the player name from the input stream matches the player name in the table entry:

for ( record in AtBats_stream where player = StatRequest.player ) {

if ( record.bases > 0 ) {

hits:= hits + 1;

}

atBats := atBats + 1;

}

This method has the advantage of reducing processing time, because the records of other players are never pulled from the reference table.

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

173

c. Explicitly create and modify the iterator to use across the table, again taking advantage of the

<AtBats_stream>

that is automatically created with the AtBats reference query:

AtBats_iterator := getIterator( AtBats_stream );

resetIterator( AtBats_iterator);

setRange( AtBats_iterator, player, StatRequest.player );

typeof(AtBats) result := getNext( AtBats_iterator );

while (not(isnull(result))) {

if ( result.bases > 0 ) {

hits := hits + 1;

}

atBats := atBats + 1;

result := getNext( AtBats_iterator );

}

This method has the advantage of giving you the most explicit control over the processing.

8.8.6 Using a Reference Table Query in CCLScript to Get a

Single Row from a Table

This example uses a reference table query within a CCLScript routine to obtain a single row from a table.

Context

In this example, we want to obtain a list of chain restaurants that have a franchise in a specified city.

Procedure

1. Obtain the necessary information about the external database table containing the row you want to obtain:

○ The name of the table containing the data

○ The schema for that table

○ The service to use to connect to it

2. Create a reference table query in your project.

In this example, the reference table name is restaurants:

CREATE REFERENCE restaurants

SCHEMA (storeID integer, city string, name string, address string)

PROPERTIES service='HANA', source='restaurants';

Because we have not specified a primary key in this reference, you can omit the sourceSchema property.

3. Create a second reference table query in your project.

In this example, the reference table query name is chainList:

CREATE REFERENCE chainList

SCHEMA (chainID integer, name string, headquarterAddress string)

174

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

PROPERTIES service='HANA', source='chainList';

4. Create the input stream for events to enter the project.

In this example, the input stream is named restaurantQuery, and consists of a single string, city:

CREATE INPUT STREAM restaurantQuery SCHEMA (city string);

5. Initialize the Flex operator, named optionQuery, to read in the reference table queries restaurants and chainLis, and restaurantQuery. Output the restaurant chain names in the output stream chainsInCity when it receives the name of a city in the restaurantQuery stream:

CREATE FLEX optionQuery

IN restaurants, chainList, restaurantQuery

OUT OUTPUT STREAM chainsInCity

SCHEMA (name string, address string)

BEGIN

ON restaurantQuery {

6. Add a CCLScript routine to produce the list of chains that have a presence in the city. For each restaurant chain in the chainList table, it grabs the first instance of that chain name in the restaurants table whose address is in the city specified in the restaurantQuery input stream and outputs the restaurant chain name and address.

for (record in chainList_stream) {

typeof(restaurants) inCity := restaurants_stream{[city=restaurantQuery.city;name=record.name;]};

output [name=inCity.name;address=restaurants.address;];

}

7. Since you are not performing any calculations at this time, close the Flex operator.

};

END;

8.9 Creating and Modifying Compute Queries

Context

The compute query creates in-line expressions in a stream or window. In the visual editor, the compute query is a shape within the palette. In the CCL editor, the compute query shape becomes a SELECT statement with column expressions.

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

P U B L I C

175

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor palette, in Streams and Windows, select Compute ( ).

3. Select a location in the diagram and click to add the shape.

4. Attach the compute object to the stream or window that provides input to this query.

Attach compute objects to any stream, window, or Flex operator. Compute objects can have only one input. Any attempt to connect more than one input source is blocked.

5. Add columns: a. Click Copy Columns from Input ( ) in the shape toolbar to copy input fields into the schema for this query.

b. Add additional columns by clicking Add Column Expression in the shape toolbar.

c. Edit a column expression by double-clicking to open the inline editor, by selecting one of the tabs in the Properties view, or by selecting an expression and pressing Ctrl+F2 to edit it using the pop-up expression editor.

Tip

When entering column names and their datatypes, use Tab to easily move between cells in the table.

6. Add column expressions , as necessary.

7. Modify column expressions by selecting and modifying them directly, or by editing the corresponding fields in the Properties view.

8. Use the toggle option to designate the compute object as INPUT or OUTPUT. By default, computes are

OUTPUT.

Example

You can also create compute queries in CCL, by using in-line expressions:

CREATE OUTPUT WINDOW Compute2 PRIMARY KEY DEDUCED KEEP 10 ROWS

SELECT

InputWindow1.Column1 Column1 ,

InputWindow1.Column2 + InputWindow1.Column3 theSum FROM InputWindow1 ;

176

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

CCL Query Construction

9 Advanced CCL Development Techniques

Use advanced CCL techniques, such as declare blocks, modules, and Flex operators, to develop sophisticated and complex projects.

● Use declare blocks to define variables, constants, CCLScript functions, and custom datatypes.

● Create modules to encapsulate reusable code.

● Use Flex operators to provide extensibility to CCL with custom event handlers, written in CCLScript.

● Use log stores to retain the contents of named windows on disk, to allow for recovery in the event of a failure.

In this section:

Declare Blocks [page 178]

Declare blocks enable you to include elements of functional programming, such as variables, parameters, typedefs, and function definitions in CCL data models.

Flex Operators [page 183]

Flex operators provide extensibility to CCL, allowing custom event handlers, written in CCLScript, to produce derived streams or windows.

Modularity [page 185]

A module in SAP HANA smart data streaming offers reusability; it can be loaded and used multiple times in a single project or in many projects.

Error Streams [page 195]

Error streams gather errors and the records that caused them.

Using Log Stores for Data Recovery [page 200]

A log store provides data recovery inside a window if a server fails or is shut down unexpectedly.

Connecting a Stream to a Derived Window [page 201]

Use a GROUP BY clause or the nextval() function to connect a stream to a derived window as part of a complex query.

Adding a Binding to a Streaming Project [page 203]

Add a binding into your project using the visual editor in the SAP HANA Streaming Development perspective.

Splitting Inputs into Multiple Outputs [page 204]

The splitter construct is a multi-way filter that sends data to different target streams depending on the filter condition. It works similar to the ANSI 'case' statement.

Setting an Aging Policy [page 205]

You can set an aging policy to flag records that have not been updated within a defined interval. This is useful for detecting records that may be stale.

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

177

9.1 Declare Blocks

Declare blocks enable you to include elements of functional programming, such as variables, parameters, typedefs, and function definitions in CCL data models.

CCL supports global and local declare blocks.

Global declare blocks

Accessible to an entire project; however, you can also set individual global declare blocks for each module.

Note

Global declare blocks are merged together if more are imported from other CCL files. Only one is allowed per project.

Local declare blocks

Declared in CREATE statements, are accessible only in the SELECT clause of the stream or window in which they are declared.

Note

The variables and functions defined in a local declare block are only accessible in the

SELECT clause and anywhere inside the Flex operator.

CCL variables allow for the storage of values that may change during the execution of the model. Variables are defined in the declare block using the CCLScript syntax.

CCL typedefs are user-defined datatypes and can also be used to create an alias for a standard datatype. Use typedef to shorten long type names. Once a typedef has been defined in the declare block, use it instead of the datatype in all CCLScript statements, and throughout the project.

CCL parameters are constants for which you can set the value at the model's runtime. Use these parameters instead of literal values in a project to allow behavior changes at runtime, such as window retention policies, store sizes, and other similar changes that can be easily modified at runtime without changing the project.

Define CCL parameters in a global declare block, and initialize them in a project configuration file. You can also set a default value for the parameter in its declaration, so that initialization at server start-up is optional.

You can create CCLScript functions in a declare block to allow for operations that are more easily handled using a procedural approach. Call these CCLScript functions from stream queries and other functions throughout the project.

178

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

9.1.1 Typedefs

Use typedefs to declare new names for existing datatypes.

Syntax

typedef existingdatatypeName newdatatypeName;

Components

Component existingdatatypeName newdatatypeName

Description

The original datatype.

The new name for the datatype.

Usage

Typedefs allow you to give new names for existing datatypes, which you can use to define new variables and parameters, and specify the return type of functions. Declare typedefs in declare blocks, UDFs and inside Flex procedures. The types declared in typedefs must resolve to simple types.

Note

For unsupported datatypes, use a typedef in a declare block to create an alias for a supported datatype.

Example

This example declares euros to be another name for the money(2) datatype: typedef money(2) euros;

Once you have defined the euro typedef, use: euros price := 10.80d2;

The example is the same as: money(2) price := 10.80d2;

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

179

9.1.2 Parameters

Parameters are constants that you set during project setup using the server-command name or the project configuration file.

Syntax

parameter typeName parameterName1 [:= constant_expression]

[,parameterName2 [:= constant_expression],…];

Components

Component typeName parameterName constant_expression

Description

The datatype of the declared parameter.

The name of the declared parameter.

An expression that evaluates to a constant.

Usage

Parameters are defined using the qualifier parameter. Optionally, you can specify a default value. The default value is used only if no value is provided for the parameter at server start-up.

Parameters can use only basic datatypes, and are declared in the global DECLARE block of a project or a module. Parameters cannot be declared with complex datatypes. Since parameters are constant, their value cannot be changed in the model.

Parameters at Project Setup

You can define parameters inside the global declare block for both a project and a module. Project-level parameters can be bound on server start-up, while module-level parameters are bound when the module is loaded.

You can assign values to parameters at server start-up time on the command line used to start the server or through the project configuration file. Provide values for any project parameters that do not have a default value. Parameters can only be bound to a new value when a module or project is loaded.

Specify a default value in the parameter declaration. The default value is used for the parameter if it is not bound to a new value when the project or module is loaded. If a parameter does not have a default value, it is bound when the module or project is loaded, or an error occurs.

180

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

When a parameter is initialized with an expression, that expression is evaluated only at compile time. The parameter is then assigned the result as its default value.

When supplying values at runtime for a parameter declared as an interval datatype, interval values are specified with the unit notation in CCL and with a bare microsecond value in the project configuration file. See the SAP HANA Smart Data Streaming: Configuration and Administration Guide for more information on project configurations and parameters in the project configuration file.

Related Information

SAP HANA Smart Data Streaming: Configuration and Administration Guide

9.1.3 Variables

Variables represent a specific piece of information that may change throughout project execution. Variables are declared using the CCLScript syntax.

Syntax

typeName {variableName[:=any_expression] [, ...]}

Usage

Variables may be declared within any declare block, CCLScript UDFs, or Flex procedures. Multiple variables may be declared on a single line.

The declaration of a variable can also include an optional initial value, which must be a constant expression.

Variables without an initial value initialize to NULL.

Variables can be of complex types. However, complex variables can only be used in local declare blocks and declare blocks within a Flex stream.

Variables declared in a local declare block may subsequently be used in SELECT clauses, but cause compiler errors when used in WHERE clauses.

Example

This example defines a variable, then uses the variable in both a regular stream and a Flex stream.

declare

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

181

INTEGER ThresholdValue := 1000; end;

// Create Schemas

Create Schema TradeSchema(

Ts bigdatetime,

Symbol STRING,

Price MONEY(4),

Volume INTEGER

);

Create Schema ControlSchema (

Msg STRING,

Value INTEGER

); //

// Input Trade Window

CREATE INPUT WINDOW TradeWindow

SCHEMA TradeSchema

PRIMARY KEY (Ts);

// Input Stream for Control Messages

CREATE INPUT STREAM ControlMsg SCHEMA ControlSchema ;

// Output window, only has rows that were greater than the thresholdvalue

// was when the row was received

CREATE Output WINDOW OutTradeWindow

SCHEMA (Ts bigdatetime, Symbol STRING, Price MONEY(4), Volume INTEGER)

PRIMARY KEY (Ts) as select *

from TradeWindow

where TradeWindow.Volume > ThresholdValue;

//Flex Stream to process the control message

CREATE FLEX FlexControlStream

IN ControlMsg

OUT OUTPUT WINDOW SimpleOutput

SCHEMA ( a integer, b string, c integer)

PRIMARY KEY ( a)

BEGIN

ON ControlMsg

{

// change the value of ThresholdValue

if ( ControlMsg.Msg = 'set') {ThresholdValue:=ControlMsg.Value;}

// The following is being populate so you can see that the

ThresholdValue is being set

output [a=ControlMsg.Value; b=ControlMsg.Msg; c=ThresholdValue; |];

}

;

END;

182

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

9.1.4 Declaring Project Variables, Parameters, Datatypes, and Functions

Declare variables, parameters, typedefs, and functions in both global and local DECLARE blocks.

Procedure

1. Declare a parameter using the SAP HANA Streaming Development perspective visual editor: a. In the Outline view, right-click Statements or one of its child folders (those directly below it) and select

Modify Edit Global Declaration(s) .

b. In the Edit Expression Value pop-up window, enter the new parameter. To see a list of datatypes, press

Ctrl+Space.

The new parameter is visible under Statements > Globals > DeclareGlobalBlock.

Note

You can double-click on either DeclareGlobalBlock or the current parameter listed to launch the

Edit Expression Value pop-up window.

2. To declare parameters, variables, or user-defined CCLScript functions using CCL: a. Create a global declare block for your project by using the DECLARE statement in your main project file.

b. Add parameters, variables, or user-defined CCLScript functions to the global declare block.

Elements defined in this declare block are accessible to any elements in the project that are not inside a module.

c. Create local declare blocks by using the DECLARE statement within derived streams, windows, or both.

d. Add variables, parameters, or user-defined CCLScript functions to the local declare block.

These elements are accessible only from within the stream, window, or Flex operator in which the block is defined.

9.2 Flex Operators

Flex operators provide extensibility to CCL, allowing custom event handlers, written in CCLScript, to produce derived streams or windows.

A Flex operator produces derived streams, windows, or keyed streams in the same way that a CREATE statement produces these elements. However, a CREATE statement uses a CCL query to derive a new window from the inputs, whereas a Flex operator uses a CCLScript script.

Flex operators make CCL extensible, allowing you to implement event processing logic that would be difficult to implement in a declarative SELECT statement. CCLScript gives you process control and provides data structures that can retain state from one event to the next.

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

183

All of the features of CCLScript are available for use in a Flex operator, including:

● Data structures

○ Variables

○ EventCache (windows)

○ Dictionaries

○ Vectors

● Control structures

○ While

○ If

○ For

A Flex operator can take any number of inputs, and they can be any mix of streams, keyed streams, or windows. You can write a CCLScript event handler for each input. When an event arrives on that input, the associated CCLScript script or method is invoked.

You do not need a method for every input. Some inputs may merely provide data for use in methods associated with other inputs; for inputs without an associated method, incoming events do not trigger an action, but are accessible to other methods in the same Flex operator.

Use multiple output statements to process an event; the outputs are collected as a transaction block.

Similarly, if a Flex operator receives a transaction block, the entire transaction block is processed and all output is collected into another transaction block. This means that downstream streams, and the record data stored within the stream, are not changed until the entire event (single event or transaction block) is processed.

In this section:

Creating a Flex Operator in the Visual Editor [page 184]

Create a Flex operator to add an event handler written in CCLScript to the project.

9.2.1 Creating a Flex Operator in the Visual Editor

Create a Flex operator to add an event handler written in CCLScript to the project.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor palette, in Streams and Windows , select Flex ( ).

3. Click anywhere in the diagram to place the Flex operator.

4. To set the name of the Flex operator, choose one of the following:

○ Click and press F2 to edit the operator name.

○ In verbose mode, click the edit

icon next to the name.

5. Connect the Flex shape to the appropriate input streams or windows.

184

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

Note

When you connect a stream or window to a Flex operator, the source is added as an input to the Flex shape by default, and an On Input method is created from the source stream or window.

6. Click Add Columns to define the schema of the events produced by the Flex operator, or set the schema to a named schema in the Properties View.

7. For each input to the Flex operator, the visual editor automatically adds a null input method. To add input methods without first connecting the Flex shape to an input, use the Add On Input Method in the shape toolbar.

Each method is a CCLScript script that is invoked when an event arrives on the associated input. In other words, these are event handlers.

a. To edit the CCLScript script for each method, select the Flex shape and press F6 to switch to the CCL editor.

The CCL editor opens with the cursor at the CREATE FLEX statement.

b. Edit the CCLScript script.

c. Press F6 to switch back to the visual editor.

8. (Optional) Add an aging policy.

9. (Optional) Click Set Output Keep Policy and set keep policy options.

9.3 Modularity

A module in SAP HANA smart data streaming offers reusability; it can be loaded and used multiple times in a single project or in many projects.

Modularity organizes project elements into self-contained, reusable components, or modules, which have welldefined inputs and outputs, and allow you to encapsulate data processing procedures that are commonly repeated.

Modules, along with other objects such as import files and the main project, have their own scope, which defines the visibility range of variables or definitions. Any variables, objects, or definitions declared in a scope are accessible within that scope only; they are inaccessible to the containing scope—called the parent scope— or to any other outer scope. The parent scope can be a module or the main project. For example, if module A loads module B and the main project loads module A, then module A's scope is the parent scope to module B.

Module A's parent scope is the main project.

Modules have explicitly declared inputs and outputs. Inputs to the module are associated with streams or windows in the parent scope, and outputs of the module are exposed to the parent scope using identifiers.

When a module is reused, any streams, variables, parameters, or other objects within the module replicate, so that each version of the module exists separately from the other versions.

You can load modules within other modules, so that module A can load module B, which can load module C, and so on. Module dependency loops are not valid. For example, if module A loads module B, which loads A, the CCL compiler generates an error indicating a dependency loop between modules A and B.

When you load a module, you can connect or bind its input streams or windows to streams in the project. The output of a module can be exposed to its parent scope and referenced in that scope using the aliases provided in the LOAD MODULE statement.

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

185

Parameters inside the module are bound to parameters in the parent scope or to constant expressions. Stores within the module are bound to stores in the parent scope. Binding a store within a module to a store outside the module means that any windows using the module store instead use the bound store.

Note

All module-related compilation errors are fatal.

In this section:

Example: Creating and Using Modules [page 187]

Use basic concepts of modularity to create a module that processes raw stock trade information and outputs a list of trades with a price exceeding 1.00.

Example: Parameters in Modules [page 188]

Develop your understanding of parameter bindings. Create a module which defines a parameter that can be bound to an expression or to another parameter in the parent scope.

Using a Module Within a Streaming Project [page 189]

Create an instance of a defined module within a streaming project, and allow the inputs and outputs of the module to be bound to streams or windows in the project.

Creating a Module in the Visual Editor [page 190]

Add a new module to an existing project in the visual editor.

Editing a Module in the Visual Editor [page 190]

Edit basic module properties and module input, output, and import functions.

Creating a Reusable Module File [page 191]

Create a new, separate module file that can be imported into a project.

Importing Definitions from Another CCL File [page 192]

Import a module file to use the module in your project.

Configuring a Loaded Module [page 193]

Add or remove input and output bindings and parameter values (if any) for a specific module instance.

Configuring a Module Repository [page 194]

Create a folder in which to store modules, and configure the smart data streaming plugin for SAP

HANA studio to use it.

Related Information

Monitoring Streams for Errors [page 197]

186

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

9.3.1 Example: Creating and Using Modules

Use basic concepts of modularity to create a module that processes raw stock trade information and outputs a list of trades with a price exceeding 1.00.

Procedure

1. Create an import file to group your schemas and allow for reuse throughout the project.

In this example, the import file is called schemas.ccl and contains:

CREATE SCHEMA TradesSchema (

Id integer,

TradeTime seconddate,

Venue string,

Symbol string,

Price float,

Shares integer

);

Note

You can define schemas directly inside a module or project; however, this example uses an import file to decrease code duplication and increase maintainability of the CCL.

2. In the project, create a module using the CREATE MODULE statement, and import the import file

(schemas.ccl) using the IMPORT statement.

CREATE MODULE FilterByPrice IN TradeData OUT FilteredTradeData

BEGIN

IMPORT 'schemas.ccl';

CREATE INPUT STREAM TradeData SCHEMA TradesSchema;

CREATE OUTPUT STREAM FilteredTradeData SCHEMA TradesSchema

AS SELECT * FROM TradeData WHERE TradeData.Price > 1.00;

END;

The module's input stream, TradeData, takes in a raw feed from the stock market, and its output stream,

FilteredTradeData, provides filtered results. Using the IMPORT statement inside the module allows you to use all of the schemas grouped in the schemas.ccl file in the module streams.

3. Load the module into your main project using the LOAD MODULE statement.

This example also shows how to connect the module to a stock market stream:

IMPORT 'schemas.ccl';

CREATE INPUT STREAM NYSEData SCHEMA TradesSchema;

LOAD MODULE FilterByPrice AS FilterOver1 IN TradeData = NYSEData OUT

FilteredTradeData = NYSEPriceOver1Data;

○ The first line of the project file imports schemas.ccl, which allows the use of the same schema as the module.

○ The input stream NYSEData represents trade information from the New York Stock Exchange.

○ The LOAD MODULE statement loads the module, FilterByPrice, which is identified by the instance name of FilterOver1.

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

187

○ Binding the module's input stream, TradeData, with the input stream NYSEData allows information to flow from the NYSEData stream into the module.

○ The output of the module is exposed to the project (NYSEPriceOver1Data).

○ To access the output of the module, select the information from the NYSEPriceOver1Data stream.

9.3.2 Example: Parameters in Modules

Develop your understanding of parameter bindings. Create a module which defines a parameter that can be bound to an expression or to another parameter in the parent scope.

The module FilterByPrice filters all incoming trades based on price, and outputs only the trades that have a price greater than the value in the minimumPrice parameter.

minimumPrice can be set when FilterByPrice is loaded, or it can be bound to another parameter within the project so that the value of minimumPrice is set when the project is loaded on the server.

The module definition is:

CREATE MODULE FilterByPrice IN TradeData OUT FilteredTradeData

BEGIN

IMPORT

'schemas.ccl';

DECLARE

parameter money(2) minimumPrice := 10.00d2;

END;

CREATE INPUT STREAM TradeData

SCHEMA TradesSchema;

CREATE OUTPUT STREAM FilteredTradeData

SCHEMA TradesSchema

AS SELECT * FROM TradeData

WHERE TradeData.Price > minimumPrice;

END;

Binding a Parameter to an Expression

In parameter-to-expression binding, minimumPrice binds to an expression at the time of loading:

LOAD MODULE FilterByPrice AS FilterOver20 IN TradeData = NYSEData OUT

FilteredTradeData = NYSEPriceOver20Data PARAMETERS minimumPrice = 20.00d2;

In this type of parameter binding, the module outputs stocks only with a price greater than 20.00.

Binding a Parameter in the Module to a Parameter in the Parent Scope

In this type of binding, the parameter inside the module binds to a parameter declared in the main project, therefore modifying the value on which trades are filtered at runtime. This is done by creating a parameter

188

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

within a DECLARE block, then binding the parameter (minimumPrice) within the module to the new parameter:

DECLARE

parameter money(2) minProjectPrice := 15.00d2;

END;

LOAD MODULE FilterByPrice

AS FilterOverMinProjPrice

IN TradeData = NYSEData

OUT FilteredTradeData = NYSEPriceOverMinProjPrice

PARAMETERS minimumPrice = minProjectPrice;

If no value is specified for the project parameter minProjectPrice at runtime, then the module is filtered based on the default value of 15.00. However, if minProjectPrice is given a value at runtime, the module filters based on that value.

No Parameter Binding

In this example, minimumPrice has a default value in the module definition, therefore no parameter binding is required when loading the module. The module can be loaded as:

LOAD MODULE FilterByPrice AS FilterOver10 IN TradeData = NYSEData OUT

FilteredTradeData = NYSEPriceOver10Data;

Since no binding is provided in the LOAD MODULE statement, the module filters on its default value of 10.00.

9.3.3 Using a Module Within a Streaming Project

Create an instance of a defined module within a streaming project, and allow the inputs and outputs of the module to be bound to streams or windows in the project.

Context

Existing modules, either created within the project or imported, can be used anywhere in a project. When you use (load) a module in a project, you attach the module inputs and outputs to streams or windows in the project by configuring bindings, and set any parameters used in the module.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the Module drawer of the visual editor palette, locate and select the module to add to the project.

The palette lists any modules defined in the current project, either in the main CCL file or in any imported

CCL files. If studio does not find any CREATE MODULE statements, the palette drawer is empty.

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

189

3. Click anywhere in the diagram to place the loaded module.

9.3.4 Creating a Module in the Visual Editor

Add a new module to an existing project in the visual editor.

Context

Create modules directly in a project when you do not plan to reuse them widely across other projects.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor palette, in Shared Components, select Module ( ).

3. Select a location in the diagram and click to add the shape.

Next Steps

Open the module to edit it by clicking the Open Module Diagram in the toolbar of the module shape. This opens a new diagram where you can add input streams or windows, simple queries, and derived streams or windows. When finished, return to the diagram that has the CREATE MODULE shape, and configure the inputs and outputs, selecting from the elements defined in the module.

9.3.5 Editing a Module in the Visual Editor

Edit basic module properties and module input, output, and import functions.

Prerequisites

Create a module.

190

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

Context

Specific module inputs and outputs are determined by project developers. Imported modules have restrictions on editing, but you can modify module inputs and outputs.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor, select the module to edit.

3. To edit the module name to be unique across all object names in the scope for this module, choose one of the following:

○ Click the module name.

○ In verbose mode, click Edit

○ Select the module, and in the Properties view modify the name value.

By default, the properties view is in the lower left of the SAP HANA Streaming Development perspective.

4. Click Add Module Exported Reference(s) ( ).

5. In the Module Exported Reference(s) dialog, select the reference(s) to add or remove, then click OK.

6. Click Add Module Inputs ( ).

7. In the Module Inputs dialog, select the inputs to add or remove, then click OK.

8. Select Add Module Outputs ( ).

9. In the Module Outputs dialog, select the outputs to add or remove, then click OK.

10. To access and edit the contents of the CREATE MODULE statement, select Open Module Diagram ( ).

11. Edit the module in the diagram that opens.

12. Add comments in the properties view.

9.3.6 Creating a Reusable Module File

Create a new, separate module file that can be imported into a project.

Context

You can create modules within a project, or in separate files that you can then import into a project. Create separate module files if you are likely to reuse a particular module often, in different projects. Module files are

CCL files hold an individual CREATE MODULE statement.

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

191

Procedure

1. Choose File New CCL Module File .

2. Enter a unique file name.

This becomes the module name, and is unique across all object names in the scope for this module.

3. (Optional) Specify a different folder.

By default, the module is created in the workspace for the current project.

4. Modify the module as required and save.

To edit the CCL, see CREATE MODULE Statement in the SAP HANA Smart Data Streaming: CCL Reference.

Related Information

SAP HANA Smart Data Streaming: CCL Reference

9.3.7 Importing Definitions from Another CCL File

Import a module file to use the module in your project.

Context

You can do this either in the CCL editor using the IMPORT statement, or by using the visual editor, as described here.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. Open the visual editor by clicking Switch to Visual, or pressing F4.

3. If Outline view is not visible, select Window Show View Outline , or press Alt+Shift+O.

4. In the Outline view, expand the Statements list.

5. Right-click the Imports statement and select Create Child Import .

6. Select the file or files to import and click OK.

7. Expand the imported file until you see the imported module.

8. Click and drag the module anywhere in the diagram.

192

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

9.3.8 Configuring a Loaded Module

Add or remove input and output bindings and parameter values (if any) for a specific module instance.

Context

Active modules are created when existing module definitions are used to create new module instances.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the diagram, select the module instance to edit.

3. To edit the name of the module instance, either:

○ Click the load module instance name.

○ In verbose mode, click Edit

4. Set the input bindings by adding connectors: a. Expand the Input Bindings compartment to that you can see the list of inputs.

b. Add connectors to the shape in the order of the list of inputs. To see the schema for an input or how a particular input is used in the module, you can look "inside" the module by clicking the Open Module

Diagram on the shape toolbar. This opens the model in a separate editor so that you can see the structure of the module.

5. Output bindings are set automatically, and the outputs appear on the diagram attached to the module instance. Rename the outputs as desired.

Note

For input bindings, the schema on both sides of the binding needs to be compatible.

6. (Optional) Modify input or output bindings by selecting an individual binding in the loaded module, and changing any of the following options in the Properties window:

Property inputStreamOrWindow streamOrWindowInModule comment (Output only) name (Output only)

Value

Select the available input stream or window components from the list.

Select the available stream or window to bind with exist ing stream or window inputs.

Add a comment or description of the output stream.

Add a name to the output stream.

7. If the module uses any parameters, parameter bindings are listed in the module instance shape on the diagram. Set parameter values in the Properties View:

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

193

parameterInModule parameterValue the parameter name.

the value to set this parameter to, for this instance of the module.

8. (Optional) Click Add Store Binding ( ). If you omit a store binding, the default memory store is used.

You can also specify a store for windows in the module.

9. Edit the store binding by selecting and modifying the available fields in the Properties window: storeInModule storeValue the classification of the string, by default NULL.

value phrase that defines the parameter binding.

10. To access input or output windows used inside a loaded module, select Open Module Diagram ( ).

9.3.9 Configuring a Module Repository

Create a folder in which to store modules, and configure the smart data streaming plugin for SAP HANA studio to use it.

Context

Modules are reusable blocks of CCL containing one or more CREATE MODULE statements. A module repository is a directory that contains these files. Once this directory has been created and configured in studio, modules can be stored in it and loaded into projects using the SAP HANA Streaming Development perspective palette.

Procedure

1. Create a new folder, or select an existing folder, to serve as the module repository.

2. In SAP HANA Streaming Development perspective, click Windows Preferences SAP HANA smart data streaming .

3. Enter the full path to the folder you want to use as the module repository in the Module Repository

Directory field.

4. Click Apply.

5. Click OK.

194

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

9.4 Error Streams

Error streams gather errors and the records that caused them.

Description

Error streams provide ways to capture error information along with the data that caused an error. This can assist in debugging errors during development, as well as provide real-time monitoring of projects in a production environment.

You can specify more than one error stream in a single project.

An error stream is identical to other user-defined streams, except it:

● Receives records from its source stream or window only when there is an error on that stream or window.

The record it receives is the input to the source stream or window that caused the error.

● Has a predefined schema that cannot be altered by the user.

The following table describes the schema for an error stream:

Column errorCode

Datatype integer errorRecord errorMessage errorTime binary string errorStreamName string sourceStreamName string bigdatetime

Description

The numeric code for the error that was reported.

The record that caused the error.

Plain text message describing the error.

The name of the stream on which this error was reported.

The name of the stream that sent the record that caused the error.

The time the error occurred; a microsecond granularity timestamp.

Error Codes and Corresponding Values

● NO_ERR - 0

● GENERIC_ERROR - 1

● FP_EXCEPTION - 2

● BADARGS - 3

● DIVIDE_BY_ZERO - 4

● OVERFLOW_ERR - 5

● UNDERFLOW_ERR - 6

● SYNTAX_ERR - 7

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

195

Limitations

The syntax of the error stream provides a mechanism for trapping runtime errors, and has these limitations:

● Only errors that occur during record computation are captured in error streams. Errors in computations that occur at server start-up, such as evaluation of expressions used to initialize variables and parameters, are not propagated to error streams. Other errors, such as connection errors and noncomputational errors, are not captured in error streams.

● Errors that occur during computations that happen without a triggering record will propagate an error record where the errorRecord field contains an empty record. Examples include the ON START TRANS and ON END TRANS blocks of the Flex block.

● For recordDataToRecord, the stream name must be a string literal constant. This allows a record type of the return value to determined during compilation.

● The triggering record must be retrieved using provided schemas. No native nested record support is provided to refer to the record directly.

● The triggering record reported is the immediate input for the stream in which the error happened. This may be a user-defined stream or an intermediate stream generated by the compiler. When using recordDataToString and recordDataToRecord, the first argument must match the intermediate stream if one has been generated.

● The subscription utility does not automatically decrypt (that is, convert from binary to ASCII) the error record.

● Output adapters do not automatically decrypt (that is, convert from binary to ASCII) the error record.

● Arithmetic and conversion errors occurring in external functions (such as C and Java) are not handled.

These errors are the user's responsibility.

● Error streams are not guaranteed to work within the debugger framework.

In this section:

Use error streams to monitor other streams for errors and the events that cause them. You can create error streams in the studio visual editor, or directly in CCL.

Examples: Error Streams in CCL [page 197]

View CCL examples of error streams in use.

Creating an Error Stream in the Visual Editor [page 198]

Add error streams using the studio visual editor.

Modifying an Error Stream [page 198]

When you are debugging a project in development or monitoring a project in production mode, you can change the specific streams that an error stream is monitoring.

Displaying Error Stream Data [page 199]

By default, error streams created in studio are output. However, if you configure error streams as local, or create them directly in CCL (where the default is local), you can change them back to output to make them visible to external subscribers. This makes real-time monitoring of the error streams possible.

196

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

9.4.1 Monitoring Streams for Errors

Use error streams to monitor other streams for errors and the events that cause them. You can create error streams in the studio visual editor, or directly in CCL.

Process

1. Identify the project and the specific streams to monitor.

2. Decide whether to use multiple error streams, then determine the visibility for each error stream.

3. Create the error streams in that project. See Creating an Error Stream in the Visual Editor [page 198] .

4. Display some or all of the information from the error streams in the error record; that is, information

aggregated or derived from the error records. See Displaying Error Stream Data [page 199]

9.4.2 Examples: Error Streams in CCL

View CCL examples of error streams in use.

In a project that has one input stream and two derived streams, create a locally visible error stream to monitor all three streams using:

CREATE ERROR STREAM AllErrors ON InputStream, DerivedStream1, DerivedStream2;

To keep a count of the errors according to the error code reported, add:

CREATE OUTPUT WINDOW errorHandlerAgg SCHEMA (errorNum integer, cnt long)

PRIMARY KEY DEDUCED

SELECT e.errorCode AS errorNum, COUNT(*) AS cnt

FROM AllErrors e

GROUP BY e.errorCode;

In a project that has three derived streams, create an externally visible error stream to monitor only the third derived stream (which calculates a volume weighted average price) using:

CREATE OUTPUT ERROR STREAM vwapErrors ON DerivedStream3;

To convert the format of the triggering record from binary to string, add:

CREATE OUTPUT STREAM vwapMessages SCHEMA (errorNum integer, streamName string, errorRecord string) AS

SELECT e.errorCode AS errorNum,

e.errorStreamName AS streamName,

recordDataToString(e.sourceStreamName, e.errorRecord) AS errorRecord

FROM vwapErrors e;

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

197

9.4.3 Creating an Error Stream in the Visual Editor

Add error streams using the studio visual editor.

Context

Whether you are debugging a project in development or monitoring a project in production mode, error streams let you see errors and the records that cause them in other streams in real time.

Note

An error stream cannot monitor other error streams.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor, open the project.

3. Click the error stream shape in the palette, then click an empty area in the diagram.

4. Click the + (plus) sign.

You see a list of streams in the project that can be monitored.

5. Specify the streams you want to monitor: click Select All or click each stream to monitor, then click OK.

The streams you specified are connected to the error stream by red lines indicating that they are sending error information.

9.4.4 Modifying an Error Stream

When you are debugging a project in development or monitoring a project in production mode, you can change the specific streams that an error stream is monitoring.

Context

Note

An error stream cannot monitor other error streams.

198

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor, open the project.

3. Locate the error stream shape in the work area and review the list of input streams.

4. Click the + (plus) sign, then click each stream to monitor, and click OK. Or, use the Connector in the palette to connect an input stream to the error stream.

A red line connects each stream to the error stream and the new stream names appear on the Inputs list.

5. (Optional) To remove input streams from the error stream, click the X in a red circle, then select each stream to remove. Click OK.

The red lines connecting the streams to the error stream and the stream names on the Inputs list are removed.

9.4.5 Displaying Error Stream Data

Context

In production mode, project monitoring may be performed externally.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor, open the project.

3. To enable real-time monitoring of errors encountered by the project, click the Type icon in the Error

Stream to toggle it from OUTPUT to LOCAL.

4. To enable ad hoc SQL queries, add a window (for example, ErrorState) to the project, downstream from the error stream.

The ErrorState window preserves the state of the error stream so it can be queried using the streamingquery utility. See streamingquery in the SAP HANA Smart Data Streaming: Utilities Guide for an example of using streamingquery to check the ErrorState.

Related Information

SAP HANA Smart Data Streaming: Utilities Guide

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

199

9.5 Using Log Stores for Data Recovery

A log store provides data recovery inside a window if a server fails or is shut down unexpectedly.

Properly specified log stores recover window elements on failure, and make sure data gets restored correctly if the server fails and restarts. You can use log stores with windows that have no retention policy; you cannot use log stores with stateless elements.

When using log stores, consider:

● Log stores only store window contents.

● Log stores do not directly store intermediate state, such as variables.

● Local Flex stream variables and data structures are not directly stored. However, they may be regenerated from source data if the source data is in persistent storage.

● Log stores do not preserve opcode information. (During periodic log store compaction and checkpointing, only the current window state is preserved. Records are then restored as inserts.)

● Row arrival order is not preserved. In any stream, multiple operations may be collapsed into a single record during log store compaction, changing arrival order. Inter-stream arrival order is not maintained.

● You can define one or more log stores in a project. When using multiple stores, make sure you prevent the occurrence of log store loops, which cause compilation errors. A log store loop is created when, for example, Window1 in Logstore1 feeds Window2 in Logstore2, which feeds Window3 in Logstore1.

● The contents of memory store windows that receive data directly from a log store window are recomputed once the log store window is restored from disk.

● The contents of memory store windows that receive data from a log store window via other memory store windows are also recomputed, once the input window's contents have been recomputed.

● In the case of partitioning, if the input of the partition target is a stream, which is a stateless element, then operations such as filter, compute, aggregate, and join are not supported.

● If the input of a partitioned target is on a memory store and the target is on a log store, this is supported only if the memory store (input element) can recover its data from an element that is on a log store.

Note

If a memory store window receives data from a log store window via a stateless element such as a stream, its contents are not restored during server recovery.

When you shut down the server normally, it performs a quiesce and checkpoint before it shuts down. It is therefore able to store all data currently in the project, as the data has been fully processed and is in a stable state. When an abnormal system shutdown occurs between checkpoints, there is no way of knowing the state of the system or where the uncheckpointed data was. Therefore, the uncheckpointed data on the input windows attached to log stores is replayed by streaming events down the project as though they were going through the input windows for the first time. The uncheckpointed data is replayed in an attempt to attain a state as close as possible to the state of smart data streaming before the abnormal shutdown.

Log stores are periodically compacted, at which point all data accumulated in the store is checkpointed and multiple operations on the same key are collapsed. After a checkpoint, the store continues appending incoming data rows to the end of the store until the next checkpoint.

200

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

Note

The recovery of data written to the store, but not yet checkpointed, is available only for input windows.

When you assign a window to a log store, you should also assign all of its input windows to a log store.

Otherwise, data written to the window after the last checkpoint is not restored.

Unlike memory stores, log stores do not extend automatically. Use the CCL maxfilesize property to specify the log store size. The size of a log store is extremely important. Log stores that are too small can cause processing to stop due to overflow, and can also cause significant performance degradation due to frequent cleaning cycles. A log store that is too large can hinder performance due to larger disk and memory requirements.

9.6 Connecting a Stream to a Derived Window

Use a GROUP BY clause or the nextval() function to connect a stream to a derived window as part of a complex query.

A derived window is a stateful element that requires either an explicit or deduced primary key. When connecting a stream to a derived window, you assign a primary key by using either a GROUP BY clause or the nextval() function. Use the GROUP BY clause to aggregate column expressions from the stream to deduce a primary key when you compile the project in studio. You cannot explicitly specify a primary key using the

GROUP BY clause. Use the nextval() function to assign an explicit primary key in the absence of a GROUP

BY clause.

When using the GROUP BY clause, you should only use additive functions, such as valueInserted(), sum(), avg(), and count(), for all column expressions other than the GROUP BY columns. Additive functions allow the server to optimize the aggregation so that additional aggregation indexes are not maintained, which avoids unwanted memory consumption, as an aggregation index holds all of the records in memory. This improves

the performance of the aggregation operation considerably. For more information, see Improving Aggregation

Performance [page 365]

. If using only additive functions is not an option, specify a retention policy other than

KEEP ALL on the stream to limit the growth of the aggregation index.

In this section:

Connecting a Stream to a Derived Window Using the GROUP BY Clause [page 202]

Use the GROUP BY clause to set a primary key for a derived window that is connected to a stream.

Connecting a Stream to a Derived Window Using nextval() [page 202]

Use the nextval() function to set a primary key for a derived window that is connected to a stream.

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

201

9.6.1 Connecting a Stream to a Derived Window Using the

GROUP BY Clause

Use the GROUP BY clause to set a primary key for a derived window that is connected to a stream.

Procedure

1. Open a new or existing smart data streaming project in studio.

2. Connect a stream to a derived window.

3. In the derived window, click Add Edit Group by Clause ( ).

4. Add the number of columns to group together. Click OK.

9.6.2 Connecting a Stream to a Derived Window Using nextval()

Use the nextval() function to set a primary key for a derived window that is connected to a stream.

Procedure

1. Open a new or existing smart data streaming project in studio.

2. Connect a stream to a derived window.

3. In the derived window under column expressions, right-click the wildcard and select Delete Element.

4. Select Copy Columns from Input to add the column expressions from the input window.

5. Click the dropdown menu for the derived window and select Modify Edit Primary Keys .

6. In the dialog box, add a column as the primary key. Click OK.

7. Right-click the expression value for the primary key column and select Edit Expression Value.

8. Delete the expression value and replace it with nextval(). Click OK.

202

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

9.7 Adding a Binding to a Streaming Project

Add a binding into your project using the visual editor in the SAP HANA Streaming Development perspective.

Prerequisites

The two streaming projects that you wish to bind together are ready for editing.

Context

There are three ways to add a binding to your project.

Procedure

● Add a binding by dragging and dropping a stream or window from the project explorer: a. Open the project to which you want to add a binding in the visual editor.

b. In your project explorer, expand the folder of the second project to bind to the first project.

c. Click the output stream or window to bind to the first project, and drag-and-drop it into the first project's diagram.

○ A new input stream or window is automatically created and added to the diagram of the first project. The new stream or window is named using the format

<First-Project-

Name>

<Stream-Or-Window-Name>

and has an inline schema that is identical to the schema of the stream/window from the second project.

○ An icon appears to the left side of the stream or window that contains the binding, signifying either an input or an output binding.

○ A new tab for the Project Configuration editor opens within the visual editor.

○ A new binding is created and automatically added to the Project Configuration (ccr) file, which contains values for the Binding name, Local stream/window name, Remote stream,

Workspace, and Project properties.

d. Use the Project Configuration editor to specify a value for the Cluster property, and optionally, values for other properties.

● Add a binding from a stream or window shape within a diagram: a. Open the project to which you want to add a binding in the visual editor.

b. In the project diagram, click the binding icon in the stream/window shape to bind to another stream or window.

A new Select Binding Target window opens.

c. Select the stream/window to bind to the first stream or window, and click OK.

○ A new tab for the Project Configuration editor opens within the visual editor.

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

203

○ An icon appears to the left side of the stream or window that contains the binding, signifying either an input or an output binding.

○ A new binding is created and automatically added to the Project Configuration (ccr) file, which contains values for the Binding name, Local stream/window name, Remote stream,

Workspace, and Project properties.

d. Use the Project Configuration editor to specify a value for the Cluster property, and optionally, values for other properties.

● Add a binding by clicking within a diagram: a. Open the project to which you want to add the binding in the visual editor.

b. Right-click any blank space in the diagram, and select Add Binding (in CCR) .

A new Binding (in CCR) window opens.

c. Select a stream or window for the Local stream/window field and a remote stream or window for the

Remote stream field.

d. Click OK.

○ A new tab for the Project Configuration editor opens within the visual editor.

○ An icon appears to the left side of the stream or window that contains the binding, signifying either an input or an output binding.

○ A new binding is created and automatically added to the Project Configuration (ccr) file, which contains values for the Binding name, Local stream/window name, Remote stream,

Workspace, and Project properties.

e. Use the Project Configuration editor to specify a value for the Cluster property, and optionally, values for other properties.

● Save your changes to the project, as well as the project configuration file.

9.8 Splitting Inputs into Multiple Outputs

The splitter construct is a multi-way filter that sends data to different target streams depending on the filter condition. It works similar to the ANSI 'case' statement.

Context

You can create a splitter to provide an operator that can split an input into multiple outputs.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor palette, under Streams and Windows, select Splitter.

3. Select a location in the diagram and click to add the shape.

204

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

4. To set the name of the splitter, either:

○ Click to edit the shape name, or, press F2.

○ In verbose mode, click the Edit icon next to the name.

5. (Optional) Click to make it an output (instead of local) if you want the splitter outputs to be visible via subscription in the runtime model.

6. Connect the splitter to a single input stream or a window.

7. (Optional) Add or remove Column Expressions for the splitter.

8. Create the splitter logic using Add When and Add Else . This creates the splitter output elements.

9. (Optional) Connect the splitter output elements of the splitter to other streams or windows.

9.9 Setting an Aging Policy

You can set an aging policy to flag records that have not been updated within a defined interval. This is useful for detecting records that may be stale.

Context

Aging policies are an advanced, optional feature for a window or other stateful elements.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the visual editor, select a window or other stateful element to which you want to add an aging policy.

3. Select Set Aging Policy and set values, which are described in the following table:

Value

Aging Time

Aging Field

(Optional) Max Aging Field Value

Description

This is an interval value. Any record in the window that has not been up dated for this much time will have the Aging Field incremented. The timer is reset when the record is updated (or the Aging Time Field changes). The pe riod can be specified in hours, minutes, seconds, milliseconds, or microsec onds.

The field in the record is incremented by 1 every time the aging time period elapses and no activity has occurred on the record, or until a maximum de fined value is reached. The default value is 1.

The maximum value that the aging field can be incremented to. If not speci fied, the aging field is incremented once.

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

P U B L I C

205

Value

(Optional) Aging Time Field

Description

The start time of the aging process. If not specified, the internal row time is used. If specified, the field must contain a valid start time.

4. (Optional) Double-click the policy to edit its parameters.

Results

When the project runs, records accumulate until the Aging Time or Max Aging Field Value is reached. On an update to a record, the age is reset to 0.

206

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Advanced CCL Development Techniques

10 Zero Data Loss

Zero data loss protects a project against data loss in the event of a client failure, server failure, or both.

Achieving zero data loss requires a judicious use of log stores when you set up the project, as well as configuration of project options and clients that use guaranteed delivery (GD).

With zero data loss:

● SAP HANA smart data streaming recovers streams or windows protected by one or more log stores to a consistent state as of the most recent checkpoint. (Any uncheckpointed data is lost and must be sent again by the publisher.)

● You can be confident you will not miss any events.

● Clients can minimize the number of duplicate events they receive by controlling how frequently they issue

GD commits.

● Publishers can ensure that the data they publish is fully processed by the server and thereby reduce transmission of duplicates when the server restarts.

● You can optionally configure the server to control how frequently it issues automatic checkpoints and thus control how much uncheckpointed data is liable to be lost on a server failure.

● At the expense of performance, you can minimize (but not fully eliminate) the production of duplicate rows on server or subscriber restart by tweaking how frequently the server checkpoints data and how frequently GD subscribers issue GD commits.

In this section:

Guaranteed Delivery and Persistent Subscribe Pattern [page 208]

Guaranteed delivery (GD) and persistent subscribe pattern (PSP) are delivery mechanisms that support high availability. They ensure that data continues to be processed from a stream or window even if the streaming server fails, the destination (third-party server) fails, or the destination does not respond for a period of time.

Consistent recovery lets you set up a project that can recover its data if it is interrupted by a server or connection failure. This feature can restore all the streams and windows in a project to a consistent state after a server or connection failure.

Zero data loss relies on data being checkpointed (that is registered and saved in project log stores).

Auto checkpoint lets you configure the checkpoint interval—the number of input transactions that triggers a checkpoint.

Bindings on Streams, Keyed Streams, and Windows [page 37] .

A lost connection or a server crash can cause data produced by a project to be lost before it is delivered to a listening client. If you cannot afford to lose data, complete the following tasks to configure zero data loss:

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

P U B L I C

207

10.1 Guaranteed Delivery and Persistent Subscribe Pattern

If possible, use guaranteed delivery rather than persistent subscribe pattern. GD uses CPU and memory resources more efficiently and is more flexible from a development standpoint, because it does not force you to decide how many subscribers will be supported when you set it up. However, you might prefer PSP if you do not want the guaranteed delivery store to be a log store for performance reasons. Using a memory store allows recovery when the client restarts, but not when the project restarts.

Guaranteed Delivery

Guaranteed delivery (GD) uses log stores to ensure that a GD subscriber registered with a GD stream or window receives all the data processed by that stream or window even if the client is not connected when the data is produced. GD is supported on streams and windows and each GD stream or window requires a log store.

You can specify GD support for a window in the Studio visual editor, or in CCL. A GD window supports multiple

GD subscribers as well as both GD and non-GD subscriptions.

To use GD:

● Assign a log store to every stream and window in the project that cannot be recovered by an upstream provider.

● Do at least one of the following:

○ Enable GD on any bindings for the project.

○ Enable GD on project adapters that support it.

○ Use the C++ SDK, the Java SDK, or the .NET SDK to configure publishers sending data to your project to retransmit any data for which they do not receive a commit confirmation.

○ Use the C++ SDK, the Java SDK, or the .NET SDK to set up GD subscriptions for client applications.

For more information, see the instructions on subscribing with guaranteed delivery in the SAP HANA

Smart Data Streaming: SDK Guide.

You can subscribe to streams and windows configured for GD using adapters, bindings, the SDKs, or the subscribe tool.

Adapters

Bindings

SDKs

Enable guaranteed delivery on the adapters that support it and configure the GD adapter parameters. See the SAP HANA Smart Data Streaming: Adapters Guide for information on adapter support for GD.

Enable GD on any project bindings to ensure that data is delivered to remote projects. See

In the SDKs, set up GD subscriptions so each client can receive data and checkpoint messages from your project. The client, in turn, periodically responds to the project server with a commit message reporting the sequence number of the latest event the client has processed. The

208

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

Tool server need not save events once it knows that all GD clients have processed them, so it can free up their space for other uses. See the SAP HANA Smart Data Streaming: SDK Guide for more information.

For testing or simple use cases, use the streamingsubscribe tool to subscribe in GD mode.

See the SAP HANA Smart Data Streaming: Utilities Guide for details.

Recommendations for Guaranteed Delivery Subscribers

Follow these recommendations to reduce your chances of receiving duplicate rows or inconsistent data after a subscriber or server restart.

● Make sure the project you subscribe to is running in consistent recovery mode. See

Consistent Recovery

[page 213]

, or topics in the SAP HANA Smart Data Streaming: Configuration and Administration Guide for details on setting project deployment options for consistent recovery.

● Subscribe to a stream or window on which GD is enabled. You can identify GD-enabled streams and windows by using:

○ The supports_gd command in the streamingprojectclient utility. See the SAP HANA Smart

Data Streaming: Utilities Guide for more information.

○ Commands in the SDKs. Ssee the SAP HANA Smart Data Streaming: SDK Guide for more information.

● Send data on to your client as it arrives or buffer it locally. Issue commits for only those messages for which you have received a checkpoint notification. If the client does not support commits, deliver only those messages that have been checkpointed by the server and cache the rest locally. This ensures that the client is always consistent with the server on a restart after a client or server failure.

● To minimize data loss, configure either:

○ The publisher to issue commits frequently. See the SAP HANA Smart Data Streaming: SDK Guide for more information.

○ The Auto Checkpoint project deployment option to control how frequently the client receives checkpoint messages. See

Adding Guaranteed Delivery [page 211]

or the SAP HANA Smart Data Streaming:

Configuration and Administration Guide for details on setting project deployment options for consistent recovery.

● When the smart data streaming server sends a checkpoint message, send a commit to the client or send the buffered rows followed by a commit.

● Issue a GD commit with the last committed sequence number to ensure that the server does not send the data again the next time the server or the subscription restarts. However, if the server does not shut down cleanly, it resends committed events that were not checkpointed.

Persistent Subscribe Pattern

Set up PSP for each subscriber individually. Enabling PSP on a data stream or window creates two new elements:

● A log window (a Flex operator with a log store)

● A truncate window (also called the control window)

The data stream or window on which PSP is enabled and the truncate window plug into the log window. The

PSP-enabled stream or window delivers data to the log window. The log window generates a sequence

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

P U B L I C

209

number, takes the opcode from the data, and places them at the beginning of each row of data. The log window sends this data to the output adapter that is attached to it. The adapter, taking advantage of guaranteed processing features on the subscribing system (where the data will be consumed), determines when to notify the truncate window that rows marked for deletion have been processed by the subscriber.

When it receives this notification, the truncate window informs the log window that the data that has been processed and the log window removes the data from its log store.

PSP-Enabled

Data Window/

Stream

Memory

Store

Log Window

Log Store

PSP-Enabled

Data Output

Window

Truncate

Window

Log Store

Figure 5: Per-Subscriber PSP Overview

Input adapters support persistent subscribe pattern (PSP) using facilities provided by the data source. Output adapters use PSP directly.

Note

The WebSphereMQ Input and Output adapters, all JMS Input and Output adapters, and the TIBCO

Rendezvous adapter all support PSP. These adapters have specific, unique PSP and GD parameters. For more information, see the SAP HANA Smart Data Streaming: Adapters Guide.

In this section:

Configure guaranteed delivery (GD) in studio for a stream or window to ensure that its output is delivered to subscribers.

Adding a Persistent Subscribe Pattern [page 212]

Set up persistent subscribe pattern (PSP) for any element in a project.

210

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

Related Information

Whenever possible, use guaranteed delivery instead of PSP. See Guaranteed Delivery and Persistent

SAP HANA Smart Data Streaming: CCL Reference

SAP HANA Smart Data Streaming: Configuration and Administration Guide

SAP HANA Smart Data Streaming: SDK Guide

SAP HANA Smart Data Streaming: Utilities Guide

SAP HANA Smart Data Streaming: Adapters Guide

10.1.1 Adding Guaranteed Delivery

Configure guaranteed delivery (GD) in studio for a stream or window to ensure that its output is delivered to subscribers.

Prerequisites

● Create at least one log store in which your guaranteed delivery stream or window can store its events and

GD logs.

● In the CCR Project Configuration editor,

○ Make sure the Consistent Recovery project deployment option is enabled. (It is enabled by default in new projects.)

○ (Optional) Enable the Auto Checkpoint project deployment option.

Context

Set up a GD stream or window using a CREATE STREAM, CREATE WINDOW, or CREATE FLEX statement.

Make sure to assign a GD log store to each GD-enabled stream or window. Do not create GD-enabled streams or windows inside modules—this is not supported because you cannot attach adapters or subscribe directly to elements in modules.

A stream or window configured for GD also supports non-GD subscriptions. Enabling GD does not significantly affect the stream or window’s performance when it has no registered GD subscribers.

You can subscribe to streams and windows configured for GD using adapters, bindings, the SDKs, or the subscribe tool.

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

P U B L I C

211

Procedure

1. Choose the stream or window to support GD.

2. (Optional) If a window that you want to support GD does not exist, create a new window: a. In the visual editor palette, in Streams and Windows, select an option to create a stream or window.

○ Streams that support GD include Input Stream, Derived Stream, and Flex.

○ Windows that support GD include Input Window, Derived Window, Flex, Aggregate, Compute, and Join.

b. Select a location in the diagram and click to add the stream or window.

3. Select the new element and look at its Properties view. (If the Properties view is not visible, from the main menu select Window Show View Properties .)

4. In the the Properties view: a. Select the Properties tab and click to check the Supports Guaranteed Delivery box.

b. From the Guaranteed Delivery Store dropdown, select the log store you created for this element’s

GD logs.

c. (For GD windows only) Select the General tab. In the Store field, enter the name of the log store you created for the GD window.

10.1.2 Adding a Persistent Subscribe Pattern

Set up persistent subscribe pattern (PSP) for any element in a project.

Context

Create a PSP for a single subscriber. Start with an element such as a stream or window. When you enable PSP on the element, two additional shapes appear in the model: a control window (also known as a truncate window), and a Flex operator with a log store.

Note

Subscribe Pattern [page 208] for more information.

Procedure

1. Open the project in the visual editor and choose the element to support PSP.

2. (Optional) If an element suitable for PSP does not exist, create a new one: a. In the visual editor palette, click a shape tool such as Streams and Windows.

b. Click the shape (element) you want to create (Input Stream, for example).

212

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

c. Select a location in the diagram and click to add the shape.

3. Right-click the shape and select Modify Add Persistent Subscribe Pattern .

4. In the Select Store dialog, select either Create New Store or an existing store, then click OK.

Studio creates two elements and attaches them to the shape you are configuring for PSP:

○ A Flex operator named

<-PSP-shape-name> _log <number> . This Flex operator is also known as the log window.

○ A truncate window named

<-PSP-shape-name> _truncate <number>

. The truncate window is sometimes called the control window.

The new truncate window and the shape you are configuring for PSP connect automatically to the new

Flex operator.

Next Steps

● To complete the PSP set-up, attach an appropriate output adapter to the Flex operator.

● Repeat the steps above to add more subscribers.

10.2 Consistent Recovery

Recovery consistency depends on guidelines for log stores:

● When consistent recovery is enabled, the server uses coordinated checkpoints to save data in log stores.

● When any log store fails to complete a checkpoint, all the log stores for that project roll back to their state as of the previous successful checkpoint. This rule ensures that even if a server or connection fails, all log stores in a project are consistent with one another. However, any input data that has not been checkpointed is not recovered upon restart.

Enabling consistent recovery has no effect if there are no log stores in the project. When you enable consistent recovery for a project, place the log stores on a shared drive where all the machines in the smart data streaming cluster can access them.

Enable consistent recovery in the project configuration (CCR file), either in studio or by manually editing the

CCR file. See Project Configuration Files in the SAP HANA Smart Data Streaming: Configuration and

Administration Guide.

In consistent recovery mode, a project treats commits issued by publishers as checkpoint requests. When the publisher receives the return of a commit from the project, it can notify its source that the data in question has been processed.

All guaranteed delivery subscribers to a stream or window stored in a log store receive checkpoint notifications. GD subscribers can use this notification as an indication that it is safe to commit data in its target.

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

P U B L I C

213

Consistent recovery works well with projects configured for cold failover if log stores are set up following the log store guidelines. When a project set for cold failover stops responding, the cluster restarts the project, typically on another host. Consistent recovery enables the restarted project to come back up to a consistent state corresponding to the last checkpoint.

Note

Avoid using consistent recovery mode in projects where active-active HA mode is also enabled. Because smart data streaming is nondeterministic, enabling consistent recovery mode on the instances of an activeactive project cannot be guaranteed to produce the same data in the primary and secondary instances if there is a failure.

When consistent recovery is not enabled (which is the default state), the project does not ensure that all the log stores recover to the same point in time after a server failure. Some log stores may recover to a checkpoint state earlier in time than other log stores because the checkpoints across log stores are not treated as an atomic operation. When there is only one log store in the project, this is not an issue.

When you use consistent recovery, the recommendation that all input windows in a project and their direct or indirect dependents be placed in the same log store no longer applies. Instead, consider using multiple log stores placed on different disks to improve performance. Using multiple log stores is possible because consistent recovery ensures that all the log stores in the project are always consistent with each other.

Guaranteed Delivery and Persistent Subscribe Pattern [page 208]

Related Information

SAP HANA Smart Data Streaming: Configuration and Administration Guide

10.3 Auto Checkpoint

Zero data loss relies on data being checkpointed (that is registered and saved in project log stores). Auto checkpoint lets you configure the checkpoint interval—the number of input transactions that triggers a checkpoint.

Checkpoints are triggered when:

● A publisher in a client application issues a commit (if the consistent recovery project option is enabled)

● The project server determines a checkpoint is required

● The project processes the number of transactions you specified in the auto checkpoint option

● The project shuts down cleanly

214

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

● The project restarts after an unexpected shutdown

Auto checkpoint lets you control how often log store checkpoints occur across all input streams and windows in the project. More frequent checkpoints mean less data is lost if the server crashes. At the maximum checkpoint frequency of every input transaction (value of 1), all input data is protected except the data from the last transaction, which might not be checkpointed before a crash. Here, input transaction refers to one record, one transaction, or one envelope published to the server and currently in flight. This applies to a single publisher; if there are multiple publishers, then the last transaction for every publisher that is in flight may be lost. When you set checkpoint frequency, you make a trade-off: with frequent checkpoints you can reduce the amount of data at risk, but performance and latency may suffer as a result. The alternative is to increase performance but risk a larger amount of data loss by setting infrequent checkpoints.

Setting auto checkpoint guarantees that a checkpoint occurs at least every

<N>

rows, where

<N>

is the checkpoint interval. The checkpoint itself may include more input rows because the system ensures that all inputs (other than the input stream that triggered the checkpoint) have consumed all the data in its input queues. The actual checkpoint may happen earlier than called for by the auto checkpoint interval if the system decides it is necessary.

When the server completes a checkpoint, it sends checkpoint messages to GD subscribers to notify them that all data up to the sequence number specified in the checkpoint message can be safely recovered by the server on restart.

Setting auto checkpoint has no effect if there are no log stores in the project. Auto checkpoint is not dependent on consistent recovery; use it with consistent recovery enabled or disabled.

Note

You should only perform one of the following:

● Enable auto checkpoint

● Configure publishers sending data to the project to issue commits, which trigger checkpoints.

Guaranteed Delivery and Persistent Subscribe Pattern [page 208]

Related Information

Adding Guaranteed Delivery [page 211]

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

P U B L I C

215

10.4 Achieving Zero Data Loss

Task For Instructions, see...

Create a project with one or more guaranteed delivery streams or windows or add a GD stream or window to an existing project. You can do this in studio or by adding CREATE STREAM, CREATE WINDOW, or CREATE FLEX statements to the project CCL file.

SAP HANA Smart Data Streaming: CCL

Reference

Set up log stores for any streams or windows in the project that cannot be recov ered by an upstream provider. Review the guidelines, restrictions, and sizing in structions for log stores to ensure that your project can accurately and com pletely recreate its data after a restart.

Creating a Log Store [page 219] and all

subsections

Variables and CCLScript data structures (dictionaries, vectors, and event ca ches) do not persist in log stores and cannot be recovered after a failure. Use these structures with log stores only when:

● You can provide logic to reconstruct the structures on restart.

● Processing is not affected if the structures are missing after a restart.

Enable guaranteed delivery on any bindings associated with GD-enabled streams or windows.

Adding Guaranteed Delivery [page 211]

Before you deploy the project, enable these options in the project configuration

(CCR) file:

● Failover

● (Optional) Auto Checkpoint

High Availability, as well as the respec tive option topics, in the SAP HANA

Smart Data Streaming: Configuration

and Administration Guide.

Note

You should enable Auto Checkpoint only if you do not configure publish ers of inbound data to trigger checkpoints by issuing commits.

● Consistent Recovery

Note

You should not enable active-active HA mode for projects where consis tent recovery is enabled. Active-active mode does not safeguard against data loss or data inconsistency when the project switches from the pri mary instance to the secondary.

216

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

Task

Enable guaranteed delivery on the project’s adapters.

Note

Some adapters do not support guaranteed delivery. See the Adapter Sum

mary in the SAP HANA Smart Data Streaming: Adapters Guide for information on GD support.

For Instructions, see...

SAP HANA Smart Data Streaming:

Adapters Guide for GD-specific adapter properties.

If auto checkpoint is not enabled, configure publishers sending data to your project to issue commits to trigger server checkpoints. When the commit call re turns, the publisher knows that the server has checkpointed the data.

SAP HANA Smart Data Streaming: SDK

Guide

Note

When a project receives data from more than one publisher, a commit issued by one publisher triggers checkpointing of data sent by all publishers.

Configure publishers to retransmit any data for which they do not receive a com mit confirmation.

Set up guaranteed delivery subscriptions for client applications that consume data from your project.To confirm that subscribers have received and processed data, configure them to send GD commit calls in response to checkpoint notifica tions.

SAP HANA Smart Data Streaming: SDK

Guide

Guaranteed Delivery and Persistent Subscribe Pattern [page 208]

Related Information

Creating a Log Store [page 219]

SAP HANA Smart Data Streaming: CCL Reference

SAP HANA Smart Data Streaming: Configuration and Administration Guide

SAP HANA Smart Data Streaming: SDK Guide

SAP HANA Smart Data Streaming: Adapters Guide

SAP HANA Smart Data Streaming: Developer Guide

Zero Data Loss

P U B L I C

217

11 Working with Stores

Set store defaults, or choose a log store or memory store to determine how data from a window is saved.

Every window is assigned to a store, which holds the retained records. By default, all windows are assigned to a memory store. You can create additional stores to add data recoverability and to optimize performance, then assign windows to specific stores.

Log Stores

Log stores are created using the CREATE LOG STORE statement. Set a log store as the default store using the

CREATE DEFAULT LOG STORE statement, which overrides the default memory store.

Memory Stores

Memory stores are created using the CREATE MEMORY STORE statement. If no default store is defined, new windows are assigned to a memory store automatically. Use the statement shown above to determine specific memory store behavior and set default store settings.

In this section:

If failover is enabled, configure a log store to capture the data that flows through a project.

Creating a Memory Store [page 229]

Create a memory store to retain the state of continuous queries in memory, from the most recent server startup.

Related Information

Stores [page 42]

218

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

11.1 Creating a Log Store

If failover is enabled, configure a log store to capture the data that flows through a project.

Prerequisites

● Determine the size of your log store.

● Review

Log Store Guidelines [page 221] .

Context

Note

Log stores do not store SAP HANA smart data streaming event logs (cluster logs, server logs, or project logs).

Create one log store per project. The preferred location for log store files is the base directory where project files are stored.

Procedure

1. Open the SAP HANA Streaming Development perspective.

2. In the CCL editor, create a log store using the CREATE LOG STORE statement:

CREATE [DEFAULT] LOG STORE storename

PROPERTIES filename='filepath'

[sync={ true | false},]

[sweepamount=size,]

[reservepct=size,]

[ckcount=size,]

[maxfilesize=filesize];

3. For the filename property, enter either a relative (preferred) or absolute file path for the location of the log store:

Option

Relative path

(recom mended)

Description

A relative path is relative to the smart data streaming sandbox base directory. Using a relative path means that your log store automatically points to the sandbox base directory. Relative paths do not point to the directory stack; this means that the path does not start with a drive letter or slash (/).

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

P U B L I C

219

Option

Absolute path

(not recom mended)

Description

Absolute paths are changed into relative paths, relative to the sandbox base directory, and inter preted as literally as possible. For Windows systems, an absolute path begins with the drive letter; on UNIX systems, the absolute path begins with a slash (/). For Windows, drive letters are turned into a folder and preserved in the sandbox base directory. For example, a sandbox base directory of work1/streaming/sandbox with log stores with filename parameters of D:/streaming/ logstore and data/logstore running in project importantp1 in workspace worksp1 in stance number 0 becomes:

○ For

D:/streaming/logstore :

○ work1/streaming/sandbox/store/d/streaming/logstore/ worksp1.importantp1.0

○ For data/logstore :

○ work1/streaming/sandbox/store/data/logstore/ worksp1.importantp1.0

The relative path location must be a shared disk accessible by all cluster nodes. The log store path is specified in the filename property within the log store definition. Both a relative path and absolute path automatically place log stores under

<SAP HANA-install-directory> /data_streaming/ <SID> /

STREAMING_SHARED/store/ <filename-property> / <workspace-name> .

<projectname> .

<instance-number>

4. Enter appropriate values for the remaining properties in the CREATE LOG STORE statement.

5. Click Compile (F7).

6. Click Run Project.

In this section:

There are several considerations to review before adding a log store to your project.

Calculate the size of the log store your project requires. Correctly sizing your log store is important because stores that are too small or large can lead to performance issues.

Set sizing parameters for a log store using a CREATE LOG STORE statement.

Working with Stores [page 218]

Specify persistence to optimize data models for maximum performance.

Task overview:

Related Information

Creating a Memory Store [page 229]

220

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

11.1.1 Log Store Guidelines

There are several considerations to review before adding a log store to your project.

General Guidelines

● Place log stores on a shared drive accessible to all the machines in the cluster.

● Keep windows that change at substantially different rates in different log stores. If a log store contains a large but nearly-static stream and a small but rapidly changing stream, each cleaning cycle must process large amounts of data from the static stream. Keeping streams separate optimizes cleaning cycles.

● Put any window into a log store that:

○ Is fed by stateless elements (streams).

○ Is fed by more than one upstream source in the project data flow. This is necessary for recovery because the arrival order of rows is not preserved.

○ Cannot produce the same result before and after a disruptive event such as a server crash, based on data replayed during the recovery process.

● Log stores use window names internally for identification. Start a new file for a log store when renaming a window to which it is attached.

● Variables and CCLScript data structures (dictionaries, vectors, and event caches) do not persist in log stores and cannot be recovered after a failure. Use these structures with log stores only when:

○ You can provide logic to reconstruct the structures on restart.

○ Processing is not affected if the structures are missing after a restart.

Guidelines for Guaranteed Delivery

All the general guidelines above apply to log stores for windows with guaranteed delivery. In addition:

● Ensure that the log store for every guaranteed delivery stream or window is large enough to accommodate the required events. If the log store runs out of room, the project server shuts down. Because copies of events are kept in the same log store the stream or window is assigned to, the size of the log store for a guaranteed delivery stream or window must be significantly larger than that for a similar stream or window without guaranteed delivery.

● Put any window on which GD is enabled, and all input windows that feed GD windows, into a log store. You can put windows located between the input and GD windows in a memory store if, upon restart, they can be reconstructed to exactly the same state they were in before the server went down. If an intermediate window cannot be reconstructed to its previous state, put it in a log store.

○ If consistent recovery is not enabled, put the GD windows and all their feeder windows into the same log store. Note, however, that placing many windows in the same log store adversely affects performance.

○ If consistent recovery is enabled, you can employ as many log stores for your GD and feeder windows as necessary.

Parent topic: Creating a Log Store [page 219]

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

P U B L I C

221

Related Information

Review Log Store Guidelines [page 221]

11.1.2 Sizing a Log Store

Calculate the size of the log store your project requires. Correctly sizing your log store is important because stores that are too small or large can lead to performance issues.

Prerequisites

Context

Start this procedure by calculating your project’s internal record size. An internal record represents a row in a window. Each row contains a fixed-size header plus a variable-size payload containing the column offsets, column data, and any optional fields. Use this formula for the calculation in the first step of this procedure:

In the formula:

● M represents the number of columns

● PS represents the primitive datatype size for each of the M columns

Primitive datatypes are the building blocks that make up more complex structures such as records, dictionaries, vectors, and event caches. The following table provides the size of primative datatypes:

Datatype

Boolean

Decimal

Integer

Long

String

Float

Size in Bytes

Notes

1 + number of characters in the string Estimate an average length

222

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

Datatype

Money(n)

SecondDate

Time

MsDate

BigDateTime

Binary

Size in Bytes

Notes

4 + number of bytes in the binary value Estimate an average length

Note

Guaranteed delivery (GD) logs hold events stored for delivery. If no GD logs are stored in the log store, you may skip the first three steps of this procedure. Instead, compute the dataSize using the Playback feature in studio or the streamingplayback utility to record and play back real data to get a better idea of the

amount of data you need to store. (See Playback View [page 129] for details on Playback or the SAP HANA

Smart Data Streaming: Utilities Guide for details on streamingplayback.) The log store reports

“liveSize” in the project log file (streamingserver~

<workspace-name> .

<projectname> .

<project-instance-number> ~_ <machine-hostname> .

<cluster-node-rpc-port> .

<logserial-number> .trc) when the project exits (with log level three or higher) or after every compaction

(with log level six or higher). Use the “liveSize” value for the dataSize referenced in step 2 of this procedure and beyond.

Procedure

1. (Optional if no GD logs are stored in the log store) For each window, calculate the size of an internal record. If the window supports GD, compute the size for the GD logs separately.

For purposes of illustration, use this schema:

CREATE SCHEMA TradesSchema AS (

TradeId LONG,

Symbol STRING,

Price MONEY(4),

Volume INTEGER,

TradeDate BIGDATETIME

); a. Using the primitive sizes from the Primitive Datatype Sizes table above, compute the column values— the total size in bytes for the datatypes in the schema. For the sample schema, assuming an average

STRING length of 4, the calculation is:

8 + (4 + 1) + 8 + 4 + 8 = 33 bytes b. Add the size of the offsets to the size of the column values. The offsets are calculated as (4 * M) where

M is the number of columns. Plugging in the sample schema’s five columns, we get:

(4 * 5) + 33 = 53 bytes

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

P U B L I C

223

c. Add the size of the row header, which is always 56 bytes:

56 + 53 = 113 bytes d. Round up to the nearest number divisible by:

○ 8 if smart data streaming is running on a 64-bit architecture

○ 4 if smart data streaming is running on a 32-bit architecture

For a 64-bit installation, use this formula, where URS is the unrounded record size value you computed in step 1:

URS + (8 - (URS modulo 8))

(For a 32-bit installation, substitute a 4 for each 8 in the formula.) Continue with the example, where smart data streaming is running on a 64-bit machine:

113 + (8 - (1)) = 120 bytes e. Label your result recordSize and make a note of it.

2. Estimate the maximum amount of data, in bytes, that you expect to collect in the log store. To do this, determine the maximum number of records each window assigned to the log store will contain. If the window supports guaranteed delivery, treat the GD logs as a separate window. For the record count, use the maximum number of uncommitted rows you expect the GD logs to contain for this window. Add 1000 to this value, since GD logs are purged only when there are at least 1000 fully committed events.

a. For each window, determine the data size by multiplying the expected record count by the recordSize you computed in step 1.

b. Sum the data size for all the windows and GD logs to get the total size of the data that will be stored in the log store. Label this value dataSize.

c. Sum the record counts for each window and GD log assigned to this log store and label that value recordCount.

3. (Optional if no GD logs are stored in the log store) To calculate the basic indexing overhead, multiply the recordCount from the previous step by 96 bytes. Add the result to the dataSize value.

4. Choose the value of the reservePct parameter. Calculate the required store size, in bytes, including the reserve, as follows, where dataSize is the value you computed in the previous step: storeBytes = dataSize * 100 / (100 - reservePct)

Round storeBytes up to the next megabyte.

5. Ensure the reserve cannot be overrun by the uncheckpointed data.

Estimate the maximum amount of uncheckpointed data that is produced when the input queues of all the streams, except source streams, are full. Count the records in the queues that are located early in the sequence together with any records they produce as they are processed through the project. Include the number of output records produced by the stream for each of its input records.

This example shows the stream queue depth set to the default of 1024, for a log that contains four streams ordered like this: source --> derived1 --> derived2 --> derived3 a. Determine the number of records produced by each stream as it consumes the contents of its queue:

○ 1024 records may end up in derived1's input queue. Assuming the queue produces one output record for one input record, it produces 1024 records.

224

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

○ 2048 records may end up in derived2's input queue (1024 that are already collected on its own queue, and 1024 more are derived1). Assuming that derived2 is a join and generates an average of 2 output records for each input record, it produces 4096 records ([1024 + 1024] * 2).

○ 5120 records may end up in derived3 (1024 from its own queue and 4096 from derived2).

Assuming a pass-through ratio of 1, derived3 produces 5120 records.

Take all branches into account when the project’s topology is not linear. The pass-through ratio may differ for data coming from the different parent streams. Add up the data from all the input paths.

Each stream has only one input queue, so its depth is fixed, regardless of how many parent streams it is connected to. However, the mix of records in each queue may vary. Assume the entire queue is composed from the records that produce that highest amount of output. Some input streams may contain static data that is loaded once and never changes during normal work. You do not need to count these inputs. In the example, derived2 is a join stream, and has static data as its second input.

b. Calculate the space required by multiplying the total number of records by the average record size of that stream.

For example, if the records in derived1 average 100 bytes; derived2, 200 bytes; and derived3,

150 bytes, the calculation is:

(1024 * 100) + (4096 * 200) + (5120 * 150) = 1,689,600

Trace the record count through the entire project, starting from the source streams down to all the streams in the log store. Add the data sized from the streams located in the log store.

c. Multiply the record count by 96 bytes to calculate the indexing overhead and add the result to the volume in bytes:

(1024 + 4096 + 5120) * 96 = 983,040

1,689,600 + 983,040 = 2,672,640

Verify that this result is no larger than one quarter of the reserve size: uncheckpointedBytes < storeBytes * (reservePct / 4) / 100

If the result is larger than one quarter of the reserve size, increase the reserve percent and repeat the store size calculation. Uncheckpointed data is mainly a concern for smaller stores. Other than through the uncheckpointed data size, this overhead does not significantly affect the store size calculation, because the cleaning cycle removes it and compacts the data.

6. When you create the log store, place the value you arrived at for your log store size here, in the CREATE

LOG STORE statement’s maxfilesize parameter.

Task overview:

Creating a Log Store [page 219]

Related Information

value, see Sizing a Log Store [page 222]

SAP HANA Smart Data Streaming: Utilities Guide

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

P U B L I C

225

11.1.3 Log Store Sizing Reference

Set sizing parameters for a log store using a CREATE LOG STORE statement.

The CREATE LOG STORE parameters described here control the size and behavior of the log store.

maxfilesize Parameter

The maximum file size is the largest size, in bytes, that the log store file is allowed to reach. To calculate this

for instructions on calculating this value.

It is important to size log stores correctly because unlike memory stores, log stores do not extend automatically. A store that is too small requires more frequent cleaning cycles, which severely degrades performance. In the worst case, the log store can overflow and cause processing to stop. A store that is too large also causes performance issues due to the larger memory and disk footprint; however, these issues are not as severe as those caused by log stores that are too small.

reservePct Parameter

The reserve is intermediate or free space maintained in every log store, which is used when the store is resized and during periodic cleaning of the store. The reservePct value is a percentage of the size of the log store.

Note

If the reserve space is too small and the project runs until the store fills with data, a resize attempt may cause the store to become wedged. This means that it cannot be resized, and the data can be extracted from it only by SAP Technical Support. It is safer to have too much reserve than too little. The default of 20 percent is adequate in most situations. Multigigabyte stores may use a reduced value as low as 10 percent.

Small stores, under 30 MB, especially those with multiple streams, may require a higher reserve (up to 40 percent). If you find that 40 percent is still not enough, increase the size of the store.

SAP HANA smart data streaming automatically estimates the required reserve size and increases the reserve if it is too small. This usually affects only small stores. It is a separate operation from resizing the log store itself, which is performed by a user.

Note

Increasing the reserve reduces the amount of space left for data. Monitor project log messages for automatic adjustments when you start a new project. You may need to increase the store size if these messages appear.

As the store runs, more records are written into it until the free space falls below the reserve, at which point the source streams are temporarily stopped, the streams quiesced, and the checkpoint and cleaning cycle are performed. Streams do not quiesce immediately, but must first process any data collected in their input queues. Any data produced during quiescence is added to the store, meaning that the reserve must be large enough to accommodate this data and still have enough space left to perform the cleaning cycle. If this data

226

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

overruns the reserve, it cannot perform the cleaning cycle and the store becomes wedged. The automatic reserve calculation does not account for data that is no checkpointed.

Log Store Size Warnings

As the amount of data in the store grows, if the free space falls below 10 percent (excluding the reserve), smart data streaming starts reporting log store is nearing capacity in the project log file

(streamingserver~

<workspace-name> .

<project-name> .

<project-instancenumber> ~_ <machine-hostname> .

<cluster-node-rpc-port> .

<log-serial-number> .trc). If the data is deleted from the store in bursts (for example, if data is collected during the day, and data older than a week is discarded at the end of the day), these messages may appear intermittently even after the old data has been flushed. As the cleaning cycle rolls over the data that has been deleted, the messages disappear.

Unless your log store is very small, these warnings appear before the store runs out of space. If you see them, stop smart data streaming when convenient, and increase the store size. Otherwise, smart data streaming aborts when the free space in the project falls below the reserve size.

Recovering from a Wedged Log Store

If a log store is sized incorrectly, the entire reserve may be used up, which causes the store to become wedged. If this happens, you cannot resize the log store or preserve the content. Delete the store files and restart smart data streaming with a clean store. If you make a backup of the store files before deleting them

SAP Technical Support may be able to extract content. Change the store size in the project, and it is resized on restart. You cannot decrease the store size. When you restart a project after resizing the store, it will likely produce project log messages about the free space being below the reserve until the cleaning cycle assimilates the newly added free space.

ckcount Parameter

The ckcount checkpointing count parameter affects the size of uncheckpointed data. This count shows the number of records that may be updated before writing the intermediate index data. Setting it to a large value amortizes the overhead over many records to make it almost constant, averaging 96 bytes per record. Setting it to a small value increases the overhead. With the count set to zero, index data is written after each transaction, and, for the single-transaction records, the overhead becomes:

96 + 32 * ceiling (log

(number_of_records_in_the_stream))

If a stream is small (for example, fewer than 1000 records), the overhead for each record is:

96 + 32 * ceiling (log

(1000)) = 96 + 32 * 10 = 416

In many cases, the record itself is smaller than its overhead of 416 bytes. Since the effect is logarithmic, large streams are not badly affected. A stream with a million records has a logarithm of 20 and incurs an overhead of 736 bytes per record. The increased overhead affects performance by writing extra data and increasing the frequency of store cleaning.

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

P U B L I C

227

If your project includes any windows configured for guaranteed delivery (GD), consider adjusting the value of ckcount to improve performance and latency.

sweepamount Parameter

The sweepamount parameter determines how much of the log store file is “swept through” during each cleaning pass. The value is between 5 percent to 20 percent of the maxfilesize parameter. A good lower bound for the sweep size is half the size of the write cache on your storage array. Usually, it indicates a sweep size of 512 to 1024 megabytes. Smaller sweep sizes minimize spikes in latency at the expense of a higher average latency. High values give low average latency with higher spikes when reclaiming space.

If the value of the sweepamount parameter is too small, the system performs excessive cleaning; in some cases, this does not allow the log store to free enough space during cleaning.

The size of the sweep is also limited by the amount of free space left in reserve at the start of the cleaning cycle. If the reserve is set lower than the sweep amount and the sweep does not encounter much dead data, the sweep stops if the relocated live data fills up the reserve. The swept newly cleaned area becomes the new reserve for the next cycle. Unless other factors override, you should keep the sweep and the reserve sizes close to each other. reservePct is specified in percent, while sweepamount is specified in megabytes.

If your project includes any windows configured for guaranteed delivery (GD), consider adjusting the value of sweepamount to improve performance and latency.

Log Store Size and File Locations

Ensure the total size of all log store files does not exceed the size of the machine’s available RAM. If this occurs, the machine takes longer to process the data, causing all monitoring tools to display low CPU utilization for each stream, and standard UNIX commands such as vmstat to display high disk usage due to system paging.

For storing data locally using log stores, you should use a high-speed storage device, for example, a redundant array of independent disks (RAID) or a storage area network (SAN), preferably with a large dynamic RAM cache. For a moderately low throughput, place backing files for log stores on single disk drives, whether SAS,

SCSI, IDE, or SATA.

Parent topic: Creating a Log Store [page 219]

Related Information

Parent topic: Creating a Log Store [page 219]

228

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

11.1.4 Log Store Optimization Techniques

Specify persistence to optimize data models for maximum performance.

● Whenever possible, create a small log store to store static (dimension) data, and one or more larger log stores for dynamic (fact) data.

● If you are using multiple log stores for larger, rapidly changing, dynamic (fact) data, try to organize the stores on different RAID volumes.

● The correct sizing of log stores is extremely important.

Related Information

Working with Stores [page 218]

11.2 Creating a Memory Store

Create a memory store to retain the state of continuous queries in memory, from the most recent server startup.

Prerequisites

To ensure optimal performance, consult with your system administrator on the type, number, and index values for memory stores.

Procedure

1. In the visual editor palette, in Shared Components, click Memory Store.

2. Select a location in the diagram and click to add the shape.

3. Connect the memory store to a window.

4. Specify a name for the store that is unique within its scope for the project or module.

5. (Optional) Click Set Store Properties and modify property values. The following table describes the memory store properties you can modify:

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

P U B L I C

229

Property

Index Size Hint (KB) (INDEXSIZE

HINT)

Index Kind (INDEXTYPE)

Description

(Optional) Determines the initial number of elements in the hash table, when us ing a hash index. The value is in units of 1024. Setting this higher consumes more memory, but reduces the chances of spikes in latency. Default is 8KB.

The type of index mechanism for the stored elements. Default is Tree.

Use Tree for binary trees. Binary trees are predictable in use of memory and consistent in speed.

Use Tree for hash tables, as hash tables are faster, but they often consume more memory.

6. (Optional) Select Default to make this the default store for the project (or module).

Task overview:

Related Information

Creating a Log Store [page 219]

230

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Working with Stores

12 Developing with CCLScript

In this section:

Writing CCLScript Routines [page 231]

Reviewing samples of CCLScript code is the best way to familiarize yourself with its constructs.

Integrating CCLScript into CCL [page 233]

CCL uses Flex operators to execute CCLScript code to process events. They have local declaration blocks, which are blocks of CCLScript function and variable declarations. They also have one method block per input stream and an optional timer block also written in CCLScript.

Using CCLScript in Projects [page 236]

Two projects demonstrate how CCLScript is used.

12.1 Writing CCLScript Routines

Reviewing samples of CCLScript code is the best way to familiarize yourself with its constructs.

These code samples show how to use CCLScript. To see projects utilizing CCLScript that you can run on your

smart data streaming, refer to Using CCLScript in Projects [page 236] .

In this section:

Internal Pulsing [page 231]

A stock market feed is a good example of several updates flowing into a stream.

Order Book [page 232]

An example inspired by stock trading is maintaining the top of an order book.

12.1.1 Internal Pulsing

A stock market feed is a good example of several updates flowing into a stream.

In this example, the stock market feed keeps the last tick for each symbol. Some of the downstream calculations might be computationally expensive, and you might not need to recalculate on every change, but, instead, recalculate only every second or every 10 seconds. How can you collect and pulse the updates so that the expensive recalculations are done periodically instead of continuously?

SAP HANA Smart Data Streaming: Developer Guide

Developing with CCLScript

P U B L I C

231

The dictionary data structure and the timer facility allow you to code internal pulsing. In this example, the stream to control is called InStream. First, define two local variables in the Flex operator: integer version := 0; dictionary(typeof(InStream), integer) versionMap;

These two variables keep a current version and a version number for each record. The CCLScript code handling events from the input stream is:

{

versionMap[InStream] := version;

}

The special timer block within the Flex operator sends the inserts and updates:

{

for (k in versionMap) {

if (version = versionMap[k])

output setOpcode(k, upsert);

}

version++;

}

You can configure the interval between runs of the timer block in numbers of seconds. Only those events with the current version get sent downstream, and the version number is incremented for the next set of updates.

This code works when InStream has only inserts and updates; you may wish to extend this code to work with deletes.

12.1.2 Order Book

An example inspired by stock trading is maintaining the top of an order book.

In this example, there is a stream called Bid of bids of stocks (the example is kept simple by not considering the offer side), with records of the following type, where Id is the key field, the field that uniquely identifies a bid:

[integer Id; | string Symbol; float Price; integer Shares; ]

Bids can be changed, so not only might the stream insert a new bid, but also update or delete a previous bid.

The goal is to output the top three highest bids any time a bid is inserted or changed for a particular stock. The type of the output where Position ranges from 1 to 3 is:

[integer Position; | string Symbol; float Price; integer Shares; ]

For example, the Bids have been:

[Id=1; | Symbol='IBM'; Price=43.11; Shares=1000; ]

[Id=2; | Symbol='IBM'; Price=43.17; Shares=900]

[Id=3; | Symbol='IBM'; Price=42.66; Shares=800]

[Id=4; | Symbol='IBM'; Price=45.81; Shares=50]

The next event is:

[Id=5; | Symbol='IBM'; Price=46.41; Shares=75]

232

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing with CCLScript

The stream should then output the following records:

[Position=1; Symbol='IBM'; | Price=46.41; Shares=75]

[Position=2; Symbol='IBM'; | Price=45.81; Shares=50]

[Position=3; Symbol='IBM'; | Price=43.17; Shares=900]

Note

The latest value appears at the top.

One way to solve this problem is with an event cache that groups by stock and orders the events by price: eventCache(Bids[Symbol], coalesce, Price desc) previous;

The following code outputs the current block of the order book, down to the level specified by the depth variable:

{

integer i := 0;

string symbol := Bids.Symbol;

while ((i < count(previous.Id)) and (i < depth) ) {

output setOpcode([ Position=i; Symbol = symbol; |

Price=nth(i,previous.Price);

Shares=nth(i,previous.Shares);

], upsert);

i++;

}

while (i < depth) {

output setOpcode([ Position=i; Symbol=symbol ], safedelete);

i++;

}

12.2 Integrating CCLScript into CCL

In this section:

Access to the Event [page 234]

When an event arrives at a Flex operator from an input stream, the method for that input stream is run.

Access to Input Windows [page 234]

Within method and timer code in Flex operators, you can examine records in any of the input windows.

Output Statement [page 235]

Typically, a Flex operator method creates one or more events in response to an event. In order to use these events to affect the store of records, and to send downstream to other streams, use the output statement.

Notes on Transactions [page 236]

A Flex operator method processes one event at a time. SAP HANA smart data streaming can, however, be fed data in transaction blocks (groups of insert, update, and delete events).

SAP HANA Smart Data Streaming: Developer Guide

Developing with CCLScript

P U B L I C

233

12.2.1 Access to the Event

When an event arrives at a Flex operator from an input stream, the method for that input stream is run.

The CCLScript code for that method has two implicitly declared variables for each input stream: one for the event and one for the old version of the event. If the input stream is named InputStream, the variables are:

● InputStream, with the type of record events from the input stream, and

● InputStream_old, with the type of record events from the input stream.

When the method for input stream is run, the variable InputStream is bound to the event that arrived from that stream. If the event is an update, the variable InputStream_old is bound to the previous contents of the record, otherwise it is null.

Note

Delete events always come populated with the data previously held in the input stream.

A Flex operator can have more than one input stream. For instance, if there is another input stream called

AnotherInput, the variables AnotherInput and AnotherInput_old are implicitly declared in the method block for InputStream. They are set to null when the method block begins, but can be assigned within the block.

12.2.2 Access to Input Windows

Within method and timer code in Flex operators, you can examine records in any of the input windows.

There are implicitly declared variables:

●

<Input-Window-Name> _stream

●

<Input-Window-Name> _iterator

The variable

<Window-Name> _stream is useful for looking up values. The <Window-Name> _iterator is less used primarily by advanced users.

In this example, you are processing events from an input stream called Trades, with the following records:

[ Symbol='T'; | Shares=10; Price=22.88; ]

You might have another input window called Earnings that contains recent earnings data, storing records:

[ Symbol='T'; Quarter="2008Q1"; | Value=10000000.00; ]

In processing events from Earnings, you can look up the most recent Trades data using:

Trades := Trades_stream[Earnings];

The record in the Trades window that has the same key field Symbol. If there is no matching record in the

Trades window, the result is null.

When processing events from the Trades window, you can look up earnings data using:

Earnings := Earnings_stream{ [ Symbol = Trades.Symbol; | ] };

234

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing with CCLScript

The syntax here uses curly braces rather than square brackets because the meaning is different. The Trades event does not have enough fields to look up a value by key in the Earnings window. In particular, it's missing the field called Quarter. The curly braces indicate "find any record in the Earnings window whose Symbol field is the same as Trades.Symbol". If there is no matching record, the result is null.

To look up more than one record, use a for loop. For example, to loop through the Earnings window to find negative earnings: for (earningsRec in Earnings_stream) {

if ( (Trades.Symbol = Earnings.Symbol) and (Earnings.Value < 0) ) {

negativeEarnings := 1;

break;

}

As with other for loops in CCLScript, the variable earningsRec is a new variable where the scope is the body of the loop. You can write this slightly more compactly: for (earningsRec in Earnings_stream where Symbol=Trades.Symbol) {

if (Earnings.Value < 0) {

negativeEarnings := 1;

break;

}

This loops only over the records in the Earnings window that have a Symbol field equal to Trades.Symbol. If you were to list the key fields in the where section, the loop runs very efficiently. Otherwise, the where form is only nominally faster than the first form.

Using a Flex operator, you can access records in the window itself. For instance, if the Flex operator is called

Flex1, you can write a loop just as you can with any of the input windows: for (rec in Flex1) {

...

}

12.2.3 Output Statement

Here is example code that breaks up an order into 10 new orders for sending downstream: integer i:= 0; while (i < 10) {

output setOpcode([Id = i; |

Shares = InStream.Shares/10;

Price = InStream.Price; ], upsert);

}

Each of these is an upsert, which is a particularly safe operation; it gets turned into an insert if no record with the key exists, and an update otherwise.

SAP HANA Smart Data Streaming: Developer Guide

Developing with CCLScript

P U B L I C

235

12.2.4 Notes on Transactions

A Flex operator method processes one event at a time. SAP HANA smart data streaming can, however, be fed data in transaction blocks (groups of insert, update, and delete events).

In such cases, the method is run on each event in the transaction block. SAP HANA smart data streaming maintains an invariant: a stream takes in a transaction block, and produces a transaction block. It's always one block in, one block out. The Flex operator pulls apart the transaction block, and runs the method on each event within the block. All of the events that output are collected together. The Flex operator then atomically applies this block to its records, and sends the block to downstream streams.

If you create a bad event in processing an event, the whole block is rejected. For example, if you try to output a record with any null key columns, the whole transaction block would be rejected: output [ | Shares = InStream.Shares; Price = InStream.Price; ];

Likewise, if you try the following implicit insert: output [Id = 4; |

Shares = InStream.Shares;

Price = InStream.Price; ];

If there is already a record in the Flex operator with Id set to 4, the block is rejected. You can get a report of bad transaction blocks by starting smart data streaming with the bad record file option. It is better to ensure that key columns are not null, and use setOpcode to create upsert or safedelete events so that the transaction block is accepted.

Transaction blocks are made as small as possible before they are sent to other streams. For instance, if your code outputs two updates with the same keys, only the second update is sent downstream. If your code outputs an insert followed by a delete, both events are removed from the transaction block. Thus, you might output many events, but the transaction block might contain only some of them.

12.3 Using CCLScript in Projects

Two projects demonstrate how CCLScript is used.

The following project displays the top three prices for each stock symbol:

CREATE SCHEMA TradesSchema (

Id integer,

TradeTime seconddate,

Venue string,

Symbol string,

Price float,

Shares integer

);

/* ******************************************************

* Create a Nasdaq Trades Input Window

CREATE INPUT WINDOW QTrades SCHEMA

TradesSchema PRIMARY KEY (Id);

/* ******************************************************

* Use Case a:

* Keep records corresponding to only the top three

236

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing with CCLScript

* distinct values. Delete records that falls of the top

* three values.

* Here the trades corresponding to the top three prices

* per Symbol is maintained. It uses

* - eventcaches

* - local UDF

CREATE FLEX Top3TradesFlex

IN QTrades

OUT OUTPUT WINDOW Top3Trades SCHEMA TradesSchema PRIMARY KEY(Symbol,Price)

BEGIN

DECLARE

eventCache(QTrades[Symbol], manual, Price asc) tradesCache;

* Inserts record into cache if in top 3 prices and returns

* the record to delete or just the current record if it was

* inserted into cache with no corresponding delete.

typeof(QTrades) insertIntoCache( typeof(QTrades) qTrades )

{

// keep only the top 3 distinct prices per symbol in the

// event cache

integer counter := 0;

typeof (QTrades) rec;

long cacheSz := cacheSize(tradesCache);

while (counter < cacheSz) {

rec := getCache( tradesCache, counter );

if( round(rec.Price,2) = round(qTrades.Price,2) ) {

// if the price is the same update

// the record.

deleteCache(tradesCache, counter);

insertCache( tradesCache, qTrades );

return rec;

break;

} else if( qTrades.Price < rec.Price) {

break;

}

counter++;

}

//Less than 3 distinct prices

if(cacheSz < 3) {

insertCache(tradesCache, qTrades);

return qTrades;

} else { //Current price is > lowest price

//delete lowest price record.

rec := getCache(tradesCache, 0);

deleteCache(tradesCache, 0);

insertCache(tradesCache, qTrades);

return rec;

}

return null;

}

END;

ON QTrades {

keyCache( tradesCache, [Symbol=QTrades.Symbol;|] );

typeof(QTrades) rec := insertIntoCache( QTrades );

if(rec.Id) {

//When id does not match current id it is a

//record to delete

if(rec.Id <> QTrades.Id) {

output setOpcode(rec, delete);

}

output setOpcode(QTrades, upsert);

}

};

SAP HANA Smart Data Streaming: Developer Guide

Developing with CCLScript

P U B L I C

237

END;

The following project collects data for thirty seconds and then computes the desired output values:

CREATE SCHEMA TradesSchema (

Id integer,

TradeTime seconddate,

Venue string,

Symbol string,

Price float,

Shares integer

);

/* ******************************************************

* Create a Nasdaq Trades Input Window

CREATE INPUT WINDOW QTrades SCHEMA

TradesSchema PRIMARY KEY (Id) ;

/* ******************************************************

* Use Case b:

* Perform a computation every N seconds for records

* arrived in the last N seconds.

* Here the Nasdaq trades data is collected for 30 seconds

* before being released for further computation.

CREATE FLEX PeriodicOutputFlex

IN QTrades

OUT OUTPUT WINDOW QTradesPeriodicOutput SCHEMA TradesSchema PRIMARY

KEY(Symbol,Price)

BEGIN

DECLARE

dictionary(typeof(QTrades), integer) cache;

END;

ON QTrades {

//Whenever a record arrives just insert into dictionary.

//The key of the dictionary is the key to the record.

cache[QTrades] := 0;

};

EVERY 30 SECONDS {

//Cycle through event cache and output all the rows

//and delete the rows.

for (rec in cache) {

output setOpcode(rec, upsert);

}

clear(cache);

};

END;

/**

* Perform a computation from the periodic output.

CREATE OUTPUT WINDOW QTradesSymbolStats

PRIMARY KEY DEDUCED

AS SELECT

q.Symbol,

MIN(q.Price) Minprice,

MAX(q.Price) MaxPrice,

sum(q.Shares * q.Price)/sum(q.Shares) Vwap,

count(*) TotalTrades,

sum(q.Shares) TotalVolume

FROM

QTradesPeriodicOutput q

GROUP BY

q.Symbol;

238

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Developing with CCLScript

13 Exposing a Streaming Project as a Web

Service

To expose a project as a Web service using one of the SAP HANA smart data streaming Web Services providers, set the Web Service Enabled option to true in the streaming project configuration (.ccr) file.

Prerequisites

● Start the Web Services Provider on which you are exposing the project.

● To run a project in a workspace other than the default, ensure that one or more connected workspaces are available.

Note

If you are using the Streaming Web Service, you do not need to set the Web Service Enabled option.

Procedure

1. Select File Open Project... and open the project you want to run.

2. Double-click the project configuration file (

<project-name> .ccr) to open the CCR Project Configuration editor.

3. Select the Advanced tab.

4. Choose a project deployment type from the Project Deployment Details window. The following table describes the project deployment type options:

Type

Non-HA

Description

Non-HA deployments create one project option item and one instance item as children under the project deploy ment item.

HA deployments create one project option item and two instance items as children under the project deployment item. HA provides for hot project failover between instan ces.

5. Set the value of the Web Service Enabled option to true in the Project Deployment Details window.

6. Save the updated ccr file.

7. For the changes to take effect, choose one of the following:

○ Use studio or streamingclusteradmin to stop and remove the project from the node, then redeploy (add) the project.

SAP HANA Smart Data Streaming: Developer Guide

Exposing a Streaming Project as a Web Service

P U B L I C

239

○ Restart the cluster on which the project runs.

8. To run the project, choose one of the following:

○ Click Run Streaming Project

in the main toolbar (in either the SAP HANA Streaming Development or the SAP HANA Streaming Run-Test perspective) to run the project in the default workspace.

○ Click the dropdown arrow next to the Run Project tool and choose Run Streaming Project in

Workspace. Then select the workspace where this project will run.

The project runs and shows results in the SAP HANA Streaming Run-Test perspective.

240

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Exposing a Streaming Project as a Web Service

14 Machine Learning with Streaming

Predictive analysis defines functions that can perform analytic algorithms. Incremental machine learning algorithms learn and update a model on the fly, so that predictions are based on a dynamic model.

Traditional supervised learning algorithms train data models based on historical, static data. In the traditional scenario, training and retraining are infrequent events that require a large pre-existing data set to be maintained. Once training is complete, a learned model is stored in a table. When new data arrives, the scoring function makes predictions based on this stored model. As patterns change, the model needs to be retrained with more historical, labelled, data to ensure the accuracy of the algorithm. In contrast, supervised learning in streaming can continuously learn as new data arrives and is labelled, thus allowing accurate scoring in real time, which adapts to changing situations.

Traditional unsupervised learning algorithms analyze a large dataset to detect hidden patterns in data, without any labels being provided to the algorithm. When new data needs to be analyzed, the entire dataset must be re-examined to determine patterns. Conversely, unsupervised learning in streaming is able to detect novel patterns in streaming data in real time without any re-analysis of previously examined data.

By combining smart data streaming with integrated machine learning algorithms, you can leverage both supervised and unsupervised learning to train models, score and cluster data all in real time with modest memory and storage requirements.

You can use these functions from within a streaming project using CCL elements. To use machine learning functions in a streaming project, you need to define a predictive analysis model, and configure machine learning function parameters. You can add, edit or delete models within SAP HANA studio, using the data services view in the SAP HANA Streaming Development perspective. Once they are configured, you can use saved models in streaming projects to run analytic algorithms on sets of incoming data.

In this section:

Workflow: Using Machine Learning with Streaming [page 242]

Create models that use machine learning functions, insert them into streaming projects, and set up the projects to apply machine learning functions to streaming data.

Model Management [page 243]

Models help set the required variables for your machine learning function, and can be reused in multiple projects.

Example: Hoeffding Tree Training and Scoring [page 257]

This example shows Hoeffding Tree training and Hoeffding Tree scoring models used in streaming projects.

Example: DenStream Clustering [page 259]

This example shows the DenStream clustering model used in a streaming project.

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

P U B L I C

241

14.1 Workflow: Using Machine Learning with Streaming

Create models that use machine learning functions, insert them into streaming projects, and set up the projects to apply machine learning functions to streaming data.

Procedure

1. In SAP HANA studio, open the SAP HANA Streaming Run-Test perspective, and connect to the streaming server. In the server view, either:

○ Select Add New Server URL and configure a new streaming server connection, or;

○ Right-click on an existing server connection and select Connect Server.

2. In the SAP HANA Streaming Development perspective, create or load an existing SAP HANA service through the data services view. See Adding a Connection to an SAP HANA Database in the SAP HANA

Smart Data Streaming: Configuration and Administration Guide.

Note

The permissions enabled on the data service also apply to all models using this data service.

3. Open the model folder of your data service: a. Right-click on the server and select Load Workspaces.

b. Right-click on the workspace and select Load Services.

c. Right-click on the data service and select Discover, then right-click on the Models folder and select

Discover again.

4. Right-click on the Models folder and select Add Model, or select an existing model to update. Fill out the

required properties and parameters. See Model Properties and Parameters Reference [page 251] .

Every kind of machine learning model requires its own specific input and output schema format. See

Hoeffding Tree Training for Classification [page 244] ,

Hoeffding Tree Scoring for Classification [page

246]

5. Open an existing streaming project, or create a new project. See Creating a Streaming Project in SAP

HANA Studio [page 93] .

6. Create an input stream using the same schema as your selected model.

Note

Column names are case-sensitive. Make sure that the schema of the input stream matches the schema in the model. If the schemas do not match, the model stream cannot read the data, and interprets events as null. See Appendix: Troubleshooting>Model Reads Rows as Null.

7. Add the model to the project using one of the following methods:

○ In the visual editor, add a Model element from the palette.

○ In the CCL editor, use a DECLARE statement in the following format:

DECLARE MODEL <model-name>

TYPE MACHINE_LEARNING

242

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

INPUT SCHEMA ( <input-schema> )

OUTPUT SCHEMA( <output-schema> )

PROPERTIES

Dataservice = ' <service-name> ' ;

8. Add a model stream to the project using one of the following methods:

○ In the visual editor, add a Model Stream element from the palette.

○ In the CCL editor, use a CREATE STREAM statement in the following format:

CREATE STREAM <model-stream-name>

EXECUTE MODEL <model-name>

FROM <input-stream-name> ;

9. In the visual editor, attach the model and input stream elements to the model stream.

10. (Optional) Create a derived stream to gather the results of the data analysis, and redirect it to the appropriate SAP HANA table.

Related Information

SAP HANA Smart Data Streaming: Configuration and Administration Guide

14.2 Model Management

Models help set the required variables for your machine learning function, and can be reused in multiple projects.

Smart data streaming supports three kinds of predictive analytics incremental machine learning functions:

● Hoeffding Tree Training for Classification, which continuously works to discover predictive relationships, even as the streaming data changes.

● Hoeffding Tree Scoring for Classification, which applies the predictive model to the data.

● DenStream Clustering, which groups and prunes data object points, based on their weighted significance.

Combining the training and scoring classification algorithms, supervised learning in smart data streaming can continuously learn as new data arrives and is labelled, thus allowing accurate scoring in real-time, which adapts to changing situations. Using the clustering algorithm, unsupervised learning in smart data streaming is able to detect novel patterns in streaming data in real-time without any re-analysis of previously examined data.

Every kind of machine learning model requires its own specific input and output schema format. See Hoeffding

Tree Training for Classification [page 244] ,

Hoeffding Tree Scoring for Classification [page 246]

, and

DenStream Clustering [page 247]

You can create models in the SAP HANA Streaming Development perspective visual editor, or you can introduce a model into CCL using the DECLARE MODEL construct:

DECLARE MODEL ModelName

TYPE MACHINE_LEARNING

INPUT SCHEMA (id integer, col1 string, col2 string)

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

P U B L I C

243

OUTPUT SCHEMA (id integer, col1 string, col2 string)

PROPERTIES

dataservice = 'service1';

To declare a model, you need a pre-configured data service. Your model uses this data service to connect to the SAP HANA database to store and retrieve data.

In this section:

Hoeffding Tree Training for Classification [page 244]

The training model is based on a classification algorithm, which uses streaming decision tree induction derived from the Hoeffding bound.

Hoeffding Tree Scoring for Classification [page 246]

Scoring functions make calculations based on training data.

DenStream Clustering [page 247]

This model uses the DenStream algorithm to discover data object clusters of arbitrary shape, handle data outliers and noise, and by pruning, maintain only necessary information for data clustering in order to limit memory consumption.

Creating a Model [page 249]

You can create reusable models for applying machine earning functions.

Updating a Model [page 250]

You can update a model definition through the SAP HANA Streaming Development perspective data services view.

Clearing or Deleting a Model [page 250]

You can clear or delete a model definition through the SAP HANA Streaming Development perspective data services view.

Model Properties and Parameters Reference [page 251]

Using studio, fill out the applicable properties and parameters when setting up a machine learning model.

Model Database Table Reference [page 254]

Machine learning models for streaming data store information in SAP HANA database tables, under the user schema.

14.2.1 Hoeffding Tree Training for Classification

The training model is based on a classification algorithm, which uses streaming decision tree induction derived from the Hoeffding bound.

The Hoeffding bound says that with a probability of 1- δ , the mean of a random variable of range R won't differ from its estimate after n independent observations by more than:

244

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

The Hoeffding bound can give consistent results regardless of the probability distribution generating observations (the number of observations required to reach certain values may differ across distributions).

The algorithm also detects concept drift, and updates models accordingly. See the Model Properties and

Parameters Reference [page 251]

for more information about each property and parameter.

The degree to which you train before beginning to score may depend on a number of factors. However, at a minimum, scoring should not begin before the first sync point. See

Example: Hoeffding Tree Training and Scoring [page 257]

[page 246] .

Input and Output Schemas

The input schema for the training algorithm must be in the format [IDS]+S, where:

● [IDS]+ specifies the data features. Each feature column can be either an integer, a double, or a string. There is no limit to the number of feature columns.

● S specifies the label. This column must be a string.

The output schema for the training algorithm must be in the format D, which displays the accuracy of the training model. The value is a number between 0 and 1, representing the percentage of accuracy. This column must be a double.

The training model has a limit for memory use, works in a limited amount of time, and is constantly ready to predict. You can create a model in the SAP HANA Streaming Development perspective. See

Creating a Model

[page 249] .

Usage

To train a model, first set up the model properties and parameters, then use the following basic CCL structure for your streaming project:

DECLARE MODEL <model-name>

TYPE MACHINE_LEARNING

INPUT SCHEMA ( <input-schema> )

OUTPUT SCHEMA ( <output-schema> )

PROPERTIES

dataservice = ' <data-service-name> ' ;

CREATE INPUT STREAM <input-stream-name>

SCHEMA ( <stream-schema> ) ;

CREATE OUTPUT STREAM <model-stream-name>

AS EXECUTE MODEL <model-name>

FROM <input-stream-name> ;

CREATE OUTPUT STREAM <output-stream-name>

AS SELECT *

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

P U B L I C

245

FROM <model-stream-name> ;

Note

Make sure the model name is unique in the model metadata table. At run time, there can be only one training stream for a given model.

See

for a use-case example.

14.2.2 Hoeffding Tree Scoring for Classification

Scoring functions make calculations based on training data.

A scoring model must reference a training model. Without trained data, scoring cannot occur. At run time, the scoring stream periodically checks whether the training model content is updated, and grabs the latest training information when it appears.

The sync point property determines when your scoring function syncs with the database to pull in trained data.

At minimum, the scoring function needs to sync with the database at the same time as, or slightly after the first training sync point. For example, if the training function syncs with the database every 1000 events, the scoring sync point should be 1000 events or higher. See the

Model Properties and Parameters Reference

[page 251] for more information about each property and parameter.

Note

While you can start scoring as soon as there is some trained data in the database, the scoring accuracy increases in proportion to the training accuracy. For the best results, wait until training accuracy reaches your desired percentage, for example, 0.95 (95%).

Input and Ouput Schemas

The input schema for the scoring algorithm must be in the format [IS][IDS]+, where:

● [IS] specifies the ID column. This column can be either an integer or a string.

● [IDS]+ specify the data features, which correspond to the referenced training model. Each feature column can be either an integer, a double, or a string.

The output schema for the scoring algorithm must be in the format [IS]SD, where:

● [IS] specifies the ID column. This column can be either an integer or a string.

● S specifies the prediction class. This column must be a string.

● D specifies the probability that the prediction class will appear. This column must be a double.

You can create a model in the SAP HANA Streaming Development perspective. See

Creating a Model [page

249] .

246

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

Usage

To score a model, first set up the model properties and parameters, then use the following basic CCL structure for your streaming project:

MODEL <model-name>

TYPE MACHINE_LEARNING

INPUT SCHEMA ( <input-schema> )

OUTPUT SCHEMA ( <output-schema> )

PROPERTIES

dataservice = ' <data-service-name> ' ;

CREATE INPUT STREAM <input-stream-name>

SCHEMA ( <stream-schema> );

CREATE OUTPUT STREAM <model-stream-name>

AS EXECUTE MODEL <model-name>

FROM <input-stream-name> ;

CREATE OUTPUT STREAM <output-stream-name>

AS SELECT *

FROM <model-stream-name> ;

Note

Make sure the model name is unique in the model metadata table.

See

Example: Hoeffding Tree Training and Scoring [page 257]

for a use-case example.

14.2.3 DenStream Clustering

The statistical properties of the incoming streaming data change over time, and in unforeseen ways. To remain accurate, the DenStream algorithm runs constant updates. Old data or clusters are less important, and outliers may evolve into clusters. We cannot keep all the data points from the stream, so instead we form coremicro-clusters by first identifying potential core-micro-clusters and outlier-micro-clusters. To accomplish this, the algorithm uses the following formula:

For potential micro-clusters, if a data point merges with no other point, its weight will gradually decay. If the weight falls below beta mu, the point becomes an outlier, and is pruned and deleted. The weight of any new cluster is small compared with existing clusters, but we must consider that an outlier-micro-cluster may grow into a potential core-micro-cluster. The total number of micro-clusters increases logarithmically with increasing stream length, but we specify an upper bound of memory cost.

There are two steps for data clustering: an online phase of micro-cluster maintenance (performing a statistical summary of the input data), and an offline phase of generating the final clusters (based on the statistical

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

P U B L I C

247

summary). The clustering condition parameter specifies when the clustering gets triggered—either periodically, by a specific number of input events, or after a length of time. See the

Model Properties and

Parameters Reference [page 251]

for more information about each property and parameter.

Input and Output Schemas

The input schema for the DenStream clustering algorithm must be in the format [IS][IDS]+, where:

● [IS] specifies the ID column. This column can be either an integer or a string.

● [IDS]+ specify the data features, which correspond to the referenced training model. Each feature column can be either an integer, a double, or a string. There is no limit to the number of feature columns.

The output schema for the DenStream clustering algorithm must be in the format [IS]ISDISS, where:

● [IS] specifies the ID column. This column can be either an integer or a string.

● I specifies the cluster ID. This column must be an integer.

● S specifies the measure of evalution. This column must be a string.

● D specifies the evaluation value. This column must be a double.

● I displays the phase of the clustering algorithm, where 0 is initialization, 1 is the offline phase, and 2 is the online phase. This column must be an integer.

● S displays the clustering results. This column must be a string.

● S displays the outlier information. This column must be a string.

You can create a model in the SAP HANA Streaming Development perspective. See

Creating a Model [page

249] .

Usage

To use a clustering model, first set up the model properties and parameters, then use the following basic CCL structure:

DECLARE MODEL <model-name>

TYPE MACHINE_LEARNING

INPUT SCHEMA ( <input-schema> )

OUTPUT SCHEMA ( <output-schema> )

PROPERTIES

dataservice = ' <data-service-name> ' ;

CREATE INPUT STREAM <input-stream-name>

SCHEMA ( <stream-schema> ) ;

CREATE OUTPUT STREAM <model-stream-name>

AS EXECUTE MODEL <model-name>

FROM <input-stream-name> ;

CREATE OUTPUT STREAM <output-stream-name>

AS SELECT *

FROM <model-stream-name> ;

248

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

Note

Make sure the model name is unique in the model metadata table. At run time, there can be only one clustering stream for a given model.

See

Example: DenStream Clustering [page 259] for a use-case example.

14.2.4 Creating a Model

You can create reusable models for applying machine earning functions.

Prerequisites

Create and connect to a data service. See Configuring External Database Access in the SAP HANA Smart Data

Streaming: Configuration and Administration Guide.

Procedure

1. In the data services view of the SAP HANA Streaming Development perspective, right-click on a data service and select Discover.

2. Right-click on the Models folder, and select Add Model.

3. In the studio properties view, fill in the model properties and parameters, if applicable.

See the

Model Properties and Parameters Reference [page 251] for more information about each

property and parameter.

Note

The Hoeffding Tree scoring function requires an existing Hoeffding Tree training model. Before defining a scoring model, create a training model using the same data service.

4. Right-click on the model you created, and select Save Model.

5. (Optional) To set a custom name, right-click on the new model and select Rename Model.

Related Information

SAP HANA Smart Data Streaming: Configuration and Administration Guide

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

P U B L I C

249

14.2.5 Updating a Model

You can update a model definition through the SAP HANA Streaming Development perspective data services view.

Context

If you update any model parameters, and then restart a project using the same model, the model overwrites any previously stored data with new content, based on the updated parameters. You can leverage this behavior to restart the training or clustering process (since updating the parameters implies that the training or the clustering process will re-initiate).

Procedure

1. In the data services view of the SAP HANA Streaming Development perspective, right-click on a data service and select Discover.

2. Right-click on the Models folder and select Discover.

3. Select a model and edit its properties and parameters in the properties view.

4. Right-click on the model in the data services view and select Save Model.

14.2.6 Clearing or Deleting a Model

You can clear or delete a model definition through the SAP HANA Streaming Development perspective data services view.

Context

There are two methods to completely clear out model content and start from scratch: clearing a model, or deleting a model. Clearing a model only removes model content and evaluation data, with the model still remaining on the data service, and the model metadata remaining in the database tables. Deleting a model removes all model metadata from the database tables, as well as its content and evaluation data, and the model disappears from the data service.

Note

If you clear a model while a project is running, the project keeps running. However, the machine learning function in the project begins learning from scratch, starting with the most recent event.

250

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

Procedure

1. In the data services view of the SAP HANA Streaming Development perspective, right-click on a data service and select Discover.

2. Right-click on the Models folder and select Discover.

3. Right-click on the model and select either Clear Model or Delete Model.

14.2.7 Model Properties and Parameters Reference

Using studio, fill out the applicable properties and parameters when setting up a machine learning model.

Properties

Fill out the following properties when setting up any machine learning model in smart data streaming.

Property

Description

Machine Learning Function

Input Schema

Description

The description of the model.

One of the available machine learning functions: Hoeffding

Tree Training, Hoeffding Tree Scoring, or DenStream Clus tering.

The schema of the input data. Requires a comma-delimited list of datatypes.

The schema is a regular expression in the following format:

● Training:

[IDS]+S

● Scoring:

[IS][IDS]+

● Custering:

[IS][IDS]+ where I is integer , D is double , and S is string .

For an in-depth explanation of each function's input schema, see the topics

Hoeffding Tree Training for Classifi cation [page 244] ,

Hoeffding Tree Training for Classifi cation [page 244] ,

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

P U B L I C

251

Property

Output Schema

Memory Quota (MB)

Sync Point

Referenced Model

Description

The desired schema for the output data. Requires a commadelimited list of datatypes.

The schema is a regular expression in the following format:

● Training:

● Scoring:

[IS]SD

● Clustering:

[IS]ISDISS where

is integer

is double

, and

is string

For an in-depth explanation of each function's output schema, see the topics

Hoeffding Tree Training for Classifi cation [page 244] ,

The maximum amount of memory that smart data stream ing can use for the model. If the model exceeds the memory quota, the application shuts down, and prompts you to ei ther increase the quota, or clear the model content.

The interval, in seconds (S) or number of rows (N), that de termines when smart data streaming backs up model con tent. If the project shuts down unexpectedly, smart data streaming uses the most recent backup to continue training, scoring, or clustering.

Each function syncs with the SAP HANA database for a slightly different purpose:

● Training models create backups on sync, which help re cover model content after an unexpected shutdown, and retrain the model.

● Scoring models create backups on sync, and gather the latest trained data.

● Clustering models create backups on sync, which help recover model content after an unexpected shutdown.

(Used only by the Hoeffding Tree scoring function) The training model to use for scoring algorithms.

Parameters

The following parameters are used solely by the Hoeffding Tree training function:

Parameter

Split Criterion

Datatype integer

Default Value

Description

The information gain.

The only supported value is

252

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

Parameter

Split Threshold

Max Bucket Size

Tau

Delta

Datatype integer integer double double

Default Value

0.05

0.01

Description

The minimum number of re cords required in the tree node before the algorithm splits the node into new branches.

The maximum amount of data allowed in a container to detect concept drift.

The value used for tie-break ing. Tau is used when the dif ference between two attrib utes is very small, and the in formation gain is too similar to otherwise differentiate be tween the attributes.

The probability that one at tribute is superior to others when the observed differ ence of information gain is greater than epsilon.

For example, when delta is

0.05, the probability is 95%.

The following parameters are used solely by the DenStream clustering function:

Parameter

INIT N

Datatype integer

Default Value

1000

Epsilon

Beta

Mu double double double

0.2

Description

The initial number of events that trigger the clustering al gorithm.

The size of the clustering neighborhood. Objects within this radius form into one object. A large epsilon may disrupt clustering.

The outlier threshold.

The average number of points in each cluster. A smaller mu implies a larger number of micro-clusters.

Note

The product of beta and mu must be greater than

Lambda double 0.25

The decay factor.

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

P U B L I C

253

Parameter

Clustering Condition

Datatype long

Max. Number of Categories integer

Default Value

1000

Description

The number of events that need to occur before cluster ing begins. Clustering reoc curs every time the events reach the specified number.

The clustering algorithm gathers events in the offline phase, and when the number reaches the specified clus tering condition, the algo rithm enters the online phase and begins clustering.

The maximum number of at tributes in a column. Setting the value to zero allows the algorithm to prepare for any number of number of col umns; however, this can im pact performance.

For example, if the column is labeled "gender", you can ex pect the values to be "male" or "female", and therefore set the maximum number of categories to 2.

Note

If the number of attrib utes exceeds the speci fied value during run time, the algorithm reallocates more memory.

14.2.8 Model Database Table Reference

Machine learning models for streaming data store information in SAP HANA database tables, under the user schema.

When you create and save a streaming model for the first time, studio creates three tables in the SAP HANA database. Consult these tables for overview information of all the models you have created. Studio makes relevant changes to the SAP HANA database tables whenever any changes occur to the models:

● If you update a model, studio also updates the model database tables to reflect the changes.

● If you clear a model, you clear the content table and evaluation table entries for that model, but not the model metadata. This way, the algorithm of the cleared model begins learning from the beginning.

254

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

● If you delete a model, you also delete all model information from the database tables, including the model metadata.

The STREAMING_ML_MODEL_METADATA_ table contains a snapshot of the model properties:

Property

NAME

DESCRIPTION

PAL_FUNCTION

INPUT_SCHEMA

OUTPUT_SCHEMA

PARAMETERS

REFERENCED_MODEL

MEMORY_QUOTA

Description

The name of the model.

The model description.

One of the available machine-learning functions: Hoeffding

Tree Training, Hoeffding Tree Scoring, or DenStream Clus tering.

The schema of the input data, presented as a comma-delim ited list of datatypes.

The schema is a regular expression in the following format:

● Training:

[IDS]+S

● Scoring:

[IS][IDS]+

● Clustering:

[IS][IDS]+ where I is integer , D is double , and S is string .

For an in-depth explanation of each function's input schema, see the topics

Hoeffding Tree Training for Classifi cation [page 244] ,

The schema of the output data, presented as a comma-de limited list of datatypes.

The schema is a regular expression in the following format:

● Training:

● Scoring:

[IS]SD

● Clustering:

[IS]ISDISS where I is integer , D is double , and S is string .

For an in-depth explanation of each function's output schema, see the topics

Getting Started [page 261]

(Used by the Hoeffding Tree training and DenStream clus tering functions) A snapshot of the model parameters.

(Used only by the Hoeffding Tree scoring function) The training model being used for scoring algorithms.

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

P U B L I C

255

Property

SYNC_POINT

Description

Each function syncs with the SAP HANA database for a slightly different purpose:

● Training models create backups on sync, which help re cover model content after an unexpected shutdown, and retrain the model.

● Scoring models create backups on sync, and gather the latest training information.

● Clustering models create backups on sync, which help recover model content after an unexpected shutdown.

The STREAMING_ML_MODEL_CONTENT_ table stores the latest model content from PAL functions, including a snapshot of the metadata table, the processed data computed from streaming events, and information on the latest updates to the model:

Field

MODEL_NAME

INDEX

IMPORTANT_METADATA

MODEL_CONTENT_VERSION

MODEL_CONTENT

LAST_CHANGE_BY

LAST_CHANGE_AT

Description

The name of the model. This value is a primary key.

The row index. If the model content is too long to hold in one row, refer to this index. This value is a primary key.

A snapshot of the latest model metadata. Since metadata changes and updates may affect the validity of model con tent, refer to this snapshot to identify the source of the changes.

The version of the model content. The version number in creases by 1 every time the algorithm modifies the model content.

The latest model content in JSON string format.

The ID of the user that edited the model most recently.

A timestamp for the last change to the model.

The STREAMING_ML_MODEL_EVALUATION_ table stores statistics about model performance:

Field

MODEL_NAME

MEASURE_NAME

MEASURE_VALUE

Description

The name of the model. This value is a primary key.

The name of a model evaluation measure (a means of statis tically evaluating the model performance). This value is a primary key.

The value of the model evaluation measure.

256

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

14.3 Example: Hoeffding Tree Training and Scoring

This example shows Hoeffding Tree training and Hoeffding Tree scoring models used in streaming projects.

The first model in this example uses the Hoeffding Tree training function to analyze streaming census data, including personal details such as age, education, and occupation, to predict a person's income level. Data like this could be very useful to a mortgage specialist, for example, in order to judge an applicant's creditworthiness and predict what kind of mortgage is the most suitable.

First, we create the model TrainingModel on the data service palservice using the following properties:

● Description: training model

● Machine-Learning Function: HoeffdingTree Training

● Input Schema: integer, string, double, string, integer, string, string, string, string, string

● Output Schema: double

● Memory Quota: 1024

● Sync Point: 5000 N

For the purpose of this example, the values under the parameters tab remain as default.

Then, in the CCL example below, we declare the model TrainingModel with its input and output schemas, and the data service name. As you can see, the input schema of the model matches the input schema of

InStream1. The other properties and parameters of the model are not visible in CCL. The model stream

NewModelStream1 pulls in data from InStream1, and applies the analytic function from the model

TrainingModel. Then, NewModelStream1 outputs data into an SAP HANA table:

DECLARE MODEL TrainingModel

TYPE MACHINE_LEARNING

INPUT SCHEMA (

Column0 INTEGER,

Column1 STRING,

Column2 DOUBLE,

Column3 STRING,

Column4 INTEGER,

Column5 STRING,

Column6 STRING,

Column7 STRING,

Column8 STRING,

Column9 STRING )

OUTPUT SCHEMA (

Column0 DOUBLE )

PROPERTIES

Dataservice = 'palservice';

CREATE INPUT STREAM InStream1

SCHEMA

age INTEGER,

workclass STRING,

fnlwgt DOUBLE,

education STRING,

numeducation INTEGER,

maritalstatus STRING,

occupation STRING,

relationship STRING,

race STRING

);

CREATE OUTPUT STREAM NewModelStream1

AS EXECUTE MODEL trainingmodel

FROM InStream1;

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

P U B L I C

257

As the training function is running, a second project leverages the training information through the Hoeffding

Tree scoring function. Using the latest model generated from training, the scoring algorithm constantly applies the most recent training data to draw out useful statistical information.

This time, we create a second model, named ScoringModel, on the data service palservice using the following properties:

● Description: scoring model

● Machine-Learning Function: HoeffdingTree Scoring

● Input Schema: integer, integer, string, double, string, integer, string, string, string, string, string

● Output Schema: integer, string, double

● Memory Quota: 1024

● Sync Point: 5000 N

● Referenced Model: TrainingModel

Scoring models have no additional mathematical parameters.

In the example below, we use a very similar setup to the training model. In this project, however, the input schemas in both Stream1 and the scoringmodel gain a leading ID column, and the model output schema gains two more columns, which provide information about model performance. As well, this project contains an optional output stream, OutStream1, to further direct the data from the scoring model stream:

MODEL ScoringModel

TYPE MACHINE_LEARNING

INPUT SCHEMA (

Column0 INTEGER ,

Column1 INTEGER ,

Column2 STRING ,

Column3 DOUBLE ,

Column4 STRING ,

Column5 INTEGER ,

Column6 STRING ,

Column7 STRING ,

Column8 STRING ,

Column9 STRING )

OUTPUT SCHEMA (

Column0 INTEGER ,

Column1 STRING ,

Column2 DOUBLE )

PROPERTIES

dataservice = 'palservice' ;

CREATE INPUT STREAM InStream1

SCHEMA (

id INTEGER,

age INTEGER,

workclass STRING,

fnlwgt DOUBLE,

education STRING,

numeducation INTEGER,

maritalstatus STRING,

occupation STRING,

relationship STRING,

race STRING

);

CREATE OUTPUT STREAM NewModelStream1

AS EXECUTE MODEL scoringmodel

FROM in1 ;

CREATE OUTPUT STREAM OutStream1

AS SELECT *

FROM modelStream1 ;

258

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

14.4 Example: DenStream Clustering

This example shows the DenStream clustering model used in a streaming project.

In this example, the clustering algorithm evaluates streaming age and income statistics to discover patterns that are not immediately apparent to the human eye. By grouping events based on these two attributes, the algorithm learns what the the most common income amounts are for various age groups, and also identifies outlying events, such as a high-income earner in an age group where income is generally low.

First, we create the model ClusteringModel on the data service palservice using the following properties:

● Description: clustering model

● Machine-Learning Function: DenStream Clustering

● Input Schema: integer, double, integer

● Output Schema: integer, integer, string, double, integer, string, string

● Memory Quota: 1024

● Sync Point: 5 N

Then, we set the parameters as follows:

● INIT N: 50

● Epsilon: 16

● Beta: 0.2

● Mu: 10

● Lambda: 0.25

● Clustering Condition: 50

● Max. Number of Categories: 0

Then, in the CCL example below, we declare the model ClusteringModel with its input and output schemas, and the data service name. As you can see, the input schema of the model matches the input schema of

InStream1. The other properties and parameters of the model are not visible in CCL. The model stream

NewModelStream1 pulls in data from InStream1, and applies the analytic function from the model

ClusteringModel. Then, OutputStream1 selects and outputs data into an SAP HANA table:

DECLARE MODEL ClusteringModel

TYPE MACHINE_LEARNING

INPUT SCHEMA (

Column0 INTEGER,

Column1 DOUBLE,

Column2 INTEGER )

OUTPUT SCHEMA (

Column0 INTEGER ,

Column1 INTEGER ,

Column2 STRING ,

Column3 DOUBLE ,

Column4 INTEGER ,

Column5 STRING ,

Column6 STRING )

PROPERTIES

Dataservice = 'palservice';

CREATE INPUT STREAM InStream1

SCHEMA (

id INTEGER,

income DOUBLE,

age INTEGER );

CREATE OUTPUT STREAM NewModelStream1

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

P U B L I C

259

AS EXECUTE MODEL ClusteringModel

FROM InStream1;

CREATE OUTPUT STREAM OutputStream1

AS SELECT *

FROM modelStream1;

260

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Machine Learning with Streaming

15 SAP PowerDesigner for SAP HANA Smart

Data Streaming

SAP HANA smart data streaming users create and manipulate the Streaming Schema using SAP

PowerDesigner®.

PowerDesigner is a powerful modeling tool. SAP HANA smart data streaming users can use it to develop physical data models as well as the logical data models that define the Streaming Schema.

In this section:

SAP PowerDesigner® is a tool for creating and manipulating Streaming Schema. Optionally, it can be used with physical data models.

Data Model [page 263]

PowerDesigner includes a logical data model for Streaming Schema and three physical data models for the SAP IQ, SAP HANA, and SAP ASE databases.

Extended Model Setup [page 269]

Your installer automatically sets up the use of extensions.

Streaming Schema Model Development [page 273]

Develop schema using the PowerDesigner extensions.

Model Generation [page 284]

Model generation with the Streaming Schema models is a critical step in ensuring the integrity of your production environment.

Impact and Lineage Analysis [page 289]

With impact and lineage analysis, you can determine the full impact of changes to any object in the integrated model.

DDL Script Generation [page 294]

The data models for the SAP IQ, HANA, and ASE databases target different databases; however, they share an almost identical structure. Modify data models by creating additional tables or columns to suit your business environment.

15.1 Getting Started

SAP PowerDesigner® is a tool for creating and manipulating Streaming Schema. Optionally, it can be used with physical data models.

This section is intended for database and application development staff, and for SAP Professional Services representatives, customer IT support, and other technical personnel who set up and administer

PowerDesigner. It includes information you need to understand, model, and modify logical schema definitions and physical database structure when developing schema.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

261

In this section:

Data Modeling Scenarios [page 262]

Integrated modeling supports efficient schema definition and database design, and consistent production deployments.

Sample PowerDesigner Project [page 262]

A sample project supports integrated modeling.

Learning More About PowerDesigner [page 263]

For more information on using PowerDesigner, press F1 to open the online help, or see the

PowerDesigner online product documentation.

15.1.1 Data Modeling Scenarios

Integrated modeling supports efficient schema definition and database design, and consistent production deployments.

Use the Streaming Schema and extensions to:

● Model schema in the Streaming Schema model, a PowerDesigner logical data model.

● Convert Streaming Schema logical data models to SAP HANA, SAP ASE, or SAP IQ physical data models.

● Convert existing SAP HANA, SAP ASE, and SAP IQ physical data models to Streaming Schema logical data models.

● Import schema definitions defined in a CCL file into Streaming Schema models.

● Export schema definitions from Streaming Schema models into a CCL file.

● Validate a model using custom checks for Streaming Schema, in addition to the standard PowerDesigner checks.

● Analyze the impact of changes to schema, a model, or a database table on all components in the integrated model.

The corresponding adapter (SAP IQ, HANA, ASE) schema must match the SAP IQ, HANA, and ASE database schema for all tables in which data is inserted. After you make changes, you can use PowerDesigner to produce a set of data definition language (DDL) statements directly from the physical data models (SAP IQ,

HANA, and ASE). PowerDesigner saves the DDL in a SQL script that you can run to generate the tables and other objects for the target databases.

DDL generation does not require use of the extended modeling feature.

15.1.2 Sample PowerDesigner Project

A sample project supports integrated modeling.

You can install a PowerDesigner sample project that includes:

● A sample Streaming Schema logical model.

● SAP IQ, SAP HANA, and SAP ASE physical data models.

262

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

In this section:

Opening the Sample Project [page 263]

Open the sample model from the sample project.

15.1.2.1 Opening the Sample Project

Open the sample model from the sample project.

Procedure

1. Choose Start Programs Sybase PowerDesigner 16 .

2. In the Welcome dialog, under Getting started, choose Open Model or Project.

If you are not a first-time user, you may see different options in the Welcome dialog, based on your previous work in PowerDesigner.

3. Browse to the sample project in %PowerDesigner 16%\Examples\Streaming\Streaming.prj and choose Open.

PowerDesigner opens a workspace for the Streaming Schema sample project.

4. Double-click the Streaming project ( ).

The sample project opens with the sample Streaming Schema model, SAP IQ model, SAP ASE model, SAP

HANA model, and the Model Relationship Diagram in the Browser view.

15.1.3 Learning More About PowerDesigner

For more information on using PowerDesigner, press F1 to open the online help, or see the PowerDesigner online product documentation.

15.2 Data Model

PowerDesigner includes a logical data model for Streaming Schema and three physical data models for the

SAP IQ, SAP HANA, and SAP ASE databases.

The indexes for both physical data models are database-specific and are defined individually. You can open, view, modify, and extend the data models using PowerDesigner.

In this section:

Streaming Schema Logical Data Model [page 264]

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

263

The Streaming Schema model represents market data in a logical data model independent of any data store.

Finding an Object in a Diagram [page 264]

Locate any object with a symbol in a diagram or among several diagrams. Objects without graphical symbols, such as domains, are not shown in diagrams.

Data Model Tables [page 265]

The following table lists all data model tables in the Market Data diagrams with their code names and descriptions.

Extensions [page 266]

Extensions (.xem files) provide a means to customize and extend PowerDesigner metaclasses, parameters, and generation. Use extended models to store additional information, or to change model behavior.

15.2.1 Streaming Schema Logical Data Model

The Streaming Schema model represents market data in a logical data model independent of any data store.

The Streaming Schema logical model represents the building of schema and the databases parsing schema and storing them.

The Streaming Schema model contains a definition for each schema. The schema definitions are contained in the Market Data diagram in the sample Streaming Schema model. Adding schema to the diagram is optional.

To create a new Streaming Schema model, you can:

● Create it from scratch using the Streaming Schema Model category.

● Create it from scratch using the StreamingSchema.xem file to extend the model during or after creation.

● Generate it from an SAP IQ, SAP HANA, or SAP ASE physical data model.

15.2.2 Finding an Object in a Diagram

Locate any object with a symbol in a diagram or among several diagrams. Objects without graphical symbols, such as domains, are not shown in diagrams.

Procedure

Right-click an object in the Browser and select Find in Diagram.

264

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

15.2.3 Data Model Tables

The following table lists all data model tables in the Market Data diagrams with their code names and descriptions.

Table name

Bond History

Bond Quote

Bond Trade

Dividend Event

Index History

Index Intraday

Mutual Fund History

Option History

Option Quote

Code

BOND_HISTORY

BOND_QUOTE

BOND_TRADE

DIVIDEND_EVENT

INDEX_HISTORY

INDEX_INTRADAY

MUTL_FUND_HIST

OPTION_HISTORY

OPTION_QUOTE

Description

Stores bond historical data, one record per each trad ing date. The data includes daily price and yield values

(open/close, high/low), trade volume (number of bonds traded), and so on, for each bond.

Stores real-time (intraday) quote data. Each quote re cord includes a yield, bid/ask price, and size (in other words, a number of bonds offered at a bid/ask price).

Stores real-time (intraday) trade data. Each trade re cord includes a bond's price and yield and a transac tion's size (number of bonds traded).

Stores information on a dividend payment event when a shareholder receives a certain payment for each share of stock owned. The dividend amount is com monly defined as a certain percentage of a share price but can also be specified as a monetary amount. The

Monetary or Percentage Indicator (MOP_INDICATOR) column indicates how the dividend amount is defined.

Stores the index’s historical data, one record per each trading date. The data includes the index’s daily val ues (open/close, high/low) and trade volume.

Stores the index’s real-time (intraday) data that shows its value movements during a trading day. Each data point includes an index value and trade volume.

Stores the historical data for a mutual fund, one re cord per each trading date. The data includes a trade date and price.

Stores the options historical data, one record per each trading date. The data includes options daily price

(open/close, high/low), trade volume (number of con tracts traded), and so on.

Stores the options real-time (intraday) quote data.

Each quote record includes a bid/ask price, size

(number of contracts offered at a bid/ask price), and so on.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

265

Table name

Option Trade

Split Event

Stock History

Stock Quote

Stock Trade

Code

OPTION_TRADE

SPLIT_EVENT

STOCK_HISTORY

STOCK_QUOTE

STOCK_TRADE

Description

Stores the options real-time (intraday) trade data.

Each trade record includes a trade's price, size (num ber of contracts traded), and so on.

Stores information on a stock split event when the number of outstanding shares of a company’s stock is increased and the price per share is simultaneously decreased so that proportionate equity of each share holder remains the same.

The split is characterized by a split factor; a factor of

0.5 indicates that the number of shares is increased two times and that the share price is decreased two times. In a less common reverse split, the number of shares is decreased and the price per share is in creased in a similar manner; a split factor of 2 indi cates that the number of shares is decreased two times and that the share price is increased two times.

Stores the stock historical data, one record per each trading date. The data includes stocks daily prices

(open/close, high/low) and trade volume (number of shares traded).

Stores the stocks' real-time (intraday) quote data.

Each quote record includes a bid/ask price and corre sponding size values (in other words, a number of shares offered at bid/ask price).

Stores the stocks' real-time (intraday) trade data.

Each trade record includes a transaction's price and size (in other words, a number of shares traded).

15.2.4 Extensions

PowerDesigner provides four .xem files:

● StreamingSchema.xem – extensions for an logical data model. Contains rules and code that let you model Streaming Schema in a PowerDesigner logical data model.

● IQ.xem – extensions for a physical data model. Contains only transformation rules needed to convert a

Streaming Schema definition to an SAP IQ table definition, in an SAP IQ model.

● ASE.xem – extensions for a physical data model. Contains only transformation rules needed to convert a

Streaming Schema definition to an ASE table definition, in an ASE model.

266

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

● HANA.xem – extensions for a physical data model. Contains only transformation rules needed to convert a

Streaming Schema definition to an SAP HANA table definition, in an SAP HANA model.

When you use the models provided with PowerDesigner, the extensions are present. When you create a new model using the Streaming Schema model category set, extensions are applied automatically.

When you create a new model without using the Streaming Schema model categories, or when you have an existing model, you can extend it using the PowerDesigner tools and Streaming extension files.

In this section:

Category Set [page 267]

You can set the Streaming category set to create any Streaming model type.

Schema Definitions [page 267]

A schema definition in the Streaming Schema model represents a data stream in SAP HANA smart data streaming.

Impact and Lineage Analysis [page 268]

PowerDesigner provides powerful tools for analyzing the dependencies between model objects.

15.2.4.1 Category Set

You can set the Streaming category set to create any Streaming model type.

The Streaming model category set includes Streaming Schema, SAP IQ, SAP HANA, and SAP ASE categories.

To create new models from this category set, enable the categories in PowerDesigner. You can either merge the Streaming categories with others that you use, or change PowerDesigner to use only the Streaming categories.

Once you set up the Streaming category set, you can create any Streaming model type and extend it with the appropriate extension.

The Streaming.mcc file, installed with the extensions, defines the Streaming categories.

15.2.4.2 Schema Definitions

A schema definition in the Streaming Schema model represents a data stream in SAP HANA smart data streaming.

The sample Streaming Schema model contains a schema definition for each market data table. You can customize any schema definition, or create a new one.

To create a new schema in the Streaming Schema model, you can either:

● Create a schema in PowerDesigner, and then generate a CCL file from it, or;

● Import schema definitions that are defined in a CCL file.

Each schema definition contains:

● Identifiers, which associate schemas with columns that are keys in the associated table.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

267

● Attributes, which associate schemas with a destination column name in the SAP IQ, SAP HANA, and SAP

ASE databases with length and precision where appropriate, lookup table and column information for columns that are foreign keys, and descriptive notes.

In this section:

Sample Schema Definition List [page 268]

Sample schema definitions correspond to the Market Data diagram provided with PowerDesigner.

While each schema appears in the SAP IQ, SAP HANA, and SAP ASE Market Data diagram, not every table in that diagram is a schema.

15.2.4.2.1 Sample Schema Definition List

Sample schema definitions correspond to the Market Data diagram provided with PowerDesigner. While each schema appears in the SAP IQ, SAP HANA, and SAP ASE Market Data diagram, not every table in that diagram is a schema.

● Bond History

● Bond Quote

● Bond Trade

● Dividend Event

● Index History

● Index Intraday

● Mutual Fund History

● Option History

● Option Quote

● Option Trade

● Split Event

● Stock History

● Stock Quote

● Stock Trade

15.2.4.3 Impact and Lineage Analysis

PowerDesigner provides powerful tools for analyzing the dependencies between model objects.

When you perform an action on a model object, in a single operation you can produce both:

Impact Analysis to analyze the effect of the action on the objects that depend on the initial object.

Lineage Analysis to identify the objects that influence the initial object.

These tools can help you answer questions such as:

● If I change the precision on a column in my SAP ASE model which I generated from the streaming schema model, what table columns in my SAP IQ or SAP HANA model must also change, and what schema are affected?

268

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

● Which schema fields influence each column in my SAP ASE, SAP HANA, and SAP IQ models?

● If I delete a column from my SAP IQ model, what is the impact on tables and columns in my SAP ASE and

SAP IQ models, and what schema definitions must change in my streaming schema model?

15.3 Extended Model Setup

Your installer automatically sets up the use of extensions.

To apply extensions automatically to new models, set up and use the Streaming Schema model category set.

To integrate existing PDMs with the Streaming model, extend the models by attaching the appropriate extension file.

In this section:

Extending an Existing Model [page 269]

Attach extensions to any SAP IQ, SAP HANA, or SAP ASE physical data model, or to a logical data model that was generated from the Streaming physical data model but not extended.

Setting Up the Model Category Set File [page 270]

Set up PowerDesigner to use the Streaming category set for new models.

Setting Datatypes for a Streaming Schema [page 271]

Manually set the datatype attribute for a Streaming Schema definition if the Streaming Datatype column in the Attributes tab of a Streaming Schema definition is empty or shows the wrong values.

15.3.1 Extending an Existing Model

Attach extensions to any SAP IQ, SAP HANA, or SAP ASE physical data model, or to a logical data model that was generated from the Streaming physical data model but not extended.

Procedure

1. Open the model you want to extend.

2. From the PowerDesigner main menu, choose Model Extended Model Definitions .

Tip

If Extended Model Definitions is not in the menu, make sure that the extensions file is unzipped in the folder where PowerDesigner is installed.

3. Click Import an Extended Model Definition .

A list shows available extensions that have not been applied to this model.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

269

4. Select the correct model extension and choose OK.

For example, to extend an ASE physical data model, choose ASE.

5. In the List of Extended Model Definitions dialog, choose OK to extend the model.

Results

PowerDesigner applies the Streaming extensions to the model. No other changes are made. For example, a generic logical data model is not transformed to a Streaming Schema model simply by adding the extensions.

15.3.2 Setting Up the Model Category Set File

Set up PowerDesigner to use the Streaming category set for new models.

Context

PowerDesigner can display only one set of categories in the New Model dialog. While not required, using the

Streaming category makes it easier to develop models for use with SAP HANA smart data streaming.

Procedure

● Decide which option you want to use to create new models:

Option

Only the installed Streaming category set

Streaming category set merged with existing categories

Neither

Action required

Change categories

Merge Streaming categories

Manually extend any models

In this section:

Merging Categories [page 271]

When you create a new model using categories, you can see the existing categories, as well as the three standard Streaming categories. You can merge existing model categories with the Streaming category.

Changing the Default Category [page 271]

Change the default category to the Streaming category, so that you can create new Streaming models.

270

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

15.3.2.1 Merging Categories

Procedure

1. Choose Tools Resources Model Category Sets .

2. From the list in the dialog, select the set you want to add to the Streaming category.

3. Click the Merge button in the toolbar.

4. Select Streaming from the list and choose OK.

15.3.2.2 Changing the Default Category

Change the default category to the Streaming category, so that you can create new Streaming models.

Procedure

1. From the PowerDesigner main menu, choose Tools General Options .

2. Under Category, select Model Creation.

3. In the Model Creation frame, with Enable categories checked, select a default category set.

4. Click OK.

15.3.3 Setting Datatypes for a Streaming Schema

Manually set the datatype attribute for a Streaming Schema definition if the Streaming Datatype column in the

Attributes tab of a Streaming Schema definition is empty or shows the wrong values.

Context

You may need to set datatypes for a logical data model you generate from a physical data model, if the generation process cannot determine how to convert the database datatype to a Streaming datatype.

Datatypes for the shipped sample model are set correctly and no further adjustments are necessary.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

271

Procedure

1. Right-click a schema definition and choose Properties.

2. Click the Attributes tab and review values in the Streaming Datatype column.

For example, in the sample model, the Bond Quote Attributes shows the following datatypes:

Attribute Name

Instrument

Quote Date

Quote Sequence Number

Quote Time

Ask Price

Ask Size

Bid Price

Bid Size

Yield

Streaming Datatype string seconddate integer msdate money(4) integer money(4) integer money(2)

If values are missing or incorrect, continue with steps 3 - 5.

3. Click Customize Columns and Filter (Ctrl+U).

4. If needed, adjust columns available for viewing: a. Unselect Data Type, Length, and Precision.

b. Select:

Name

Streaming Datatype

Length

Precision

Mandatory

Primary Identifier

Displayed (selected by default)

5. Use the controls below the list to adjust the order so that Primary Identifier and Displayed are the last two checkboxes.

Results

Performing this task once corrects the datatypes for all schema definitions.

272

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

15.4 Streaming Schema Model Development

Develop schema using the PowerDesigner extensions.

You can:

● Explore the sample model

● Create a schema model using categories, or by creating and extending a logical data model

● Add schema to models

● Validate your schema with built-in checks, as well as custom ones

● Import defined schema definitions into a Streaming Schema model from CCL files

● Export schema definitions from the Streaming Schema model into CCL files

In this section:

Exploring the Sample Model [page 273]

Review the sample model from the sample project.

The Sample Model [page 274]

The sample model includes sample Streaming Schema, and the SAP IQ, SAP HANA, and SAP ASE data models.

Creating a Streaming Schema Model [page 274]

Create a new Streaming Schema model using the Streaming Schema category, either by creating a logical model and extending it, or by generating it from an SAP IQ, SAP HANA, or SAP ASE model that has been extended.

Validating a Model [page 281]

Check the validity of your model after schema changes, and before generating schema templates, code, or a physical model. You can check the validity of a model at any time.

Importing a CCL File [page 283]

Import the defined schema definitions in a CCL file into a Streaming Schema model.

Exporting a CCL File [page 284]

Export all the defined schema from the Streaming Schema model into a CCL file for compiling and further analysis.

15.4.1 Exploring the Sample Model

Review the sample model from the sample project.

Prerequisites

Install the sample model and complete the extended model setup.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

273

Procedure

1. Start PowerDesigner and open the sample project with the sample model.

2. To open any of the models, either:

○ Double-click the model in the Model Relationship Diagram, or,

○ In the Browser tree, double-click the model, or right-click and choose Open or Open as read-only.

Note

Do not save changes to the installed sample model. Save it to another folder so that a new version of the model and project are created.

3. To display the sample schema definitions in the Streaming Schema model, expand the navigation buttons in the Browser tree.

4. To see more information on a schema definition:

○ Right-click the schema definition, an identifier, or an attribute in the tree view and choose Properties, or,

○ Right-click the schema definition in the tree view and choose Find in Diagram.

Explore the SAP IQ, SAP HANA, and SAP ASE models in the same way.

15.4.2 The Sample Model

The sample model includes sample Streaming Schema, and the SAP IQ, SAP HANA, and SAP ASE data models.

15.4.3 Creating a Streaming Schema Model

In this section:

Creating a Model Using Categories [page 275]

Use PowerDesigner to create and automatically extend any Streaming Schema model type.

Creating a Logical Data Model [page 275]

Create a logical data model and add extensions to it.

Adding Schema Definition [page 276]

Add a schema definition by creating it, importing schema definitions in a CCL file, or generating it from an SAP IQ, SAP HANA, or SAP ASE table.

Defining Schema Properties [page 278]

Define schema details in the properties sheet.

274

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

15.4.3.1 Creating a Model Using Categories

Use PowerDesigner to create and automatically extend any Streaming Schema model type.

Prerequisites

Designate the Streaming Schema set as the default category.

Procedure

1. Choose File New Model .

2. In the New Model dialog, select Categories, and choose a category item:

○ Streaming Schema

○ SAP IQ

○ ASE

○ HANA

3. Enter a model name.

4. Click OK.

15.4.3.2 Creating a Logical Data Model

Create a logical data model and add extensions to it.

Procedure

1. Choose File New Model .

2. In the New Model dialog, select Model types and Logical Data Model.

3. Enter a model name.

4. Click the Select Extensions button to the right of the Extensions box.

A dialog shows currently loaded extensions. You can apply extensions when you create the model or later.

5. Select Streaming Schema, then select whether to share or copy:

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

275

Option

Share the extended model definitions

Copy the extended model definitions

Description

PowerDesigner always uses the contents of the .xem

file. If the contents of the .xem

file change, the model sees those changes. For example, if a future version of Streaming

Schema includes a new version of the file, models that share it sees those changes immedi ately.

Copies the contents of the .xem

file into the model. The model uses its local copy instead of the file on disk.

6. Click OK

With either approach, use other extensions besides the shipped Streaming Schema extensions by creating your own .xem file. Although it is possible to do this by adding to the StreamingSchema.xem file, SAP does not recommend this.

15.4.3.3 Adding Schema Definition

Add a schema definition by creating it, importing schema definitions in a CCL file, or generating it from an SAP

IQ, SAP HANA, or SAP ASE table.

In this section:

Creating Schema from the Schema Definitions Container [page 276]

Create a new schema definition with initial properties.

Creating Schema with the Entity Tool [page 277]

Create schema from the diagram.

Creating a Schema from the Streaming Schema Container [page 277]

Create a new schema definition with initial properties.

Generating Schema from an SAP IQ, SAP HANA, or SAP ASE Table [page 277]

Follow the same steps as when generating a Streaming Schema model, selecting a single table to generate.

15.4.3.3.1 Creating Schema from the Schema Definitions

Container

Create a new schema definition with initial properties.

Procedure

1. Open the PowerDesigner model.

2. In the Browser tree, right-click the Streaming Schema container and choose New.

3. Complete the information in the General tab or other tabs.

You can complete schema definition properties at any time before generating the physical models.

276

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

4. Click OK to save the schema definition.

15.4.3.3.2 Creating Schema with the Entity Tool

Create schema from the diagram.

Procedure

1. Open the PowerDesigner model.

2. In the diagram, click the Entity tool .

A new, empty schema definition appears in the diagram, and in the Browser tree when expanded.

3. Right-click the diagram and choose Properties.

4. Add attributes and identifiers in the properties sheet.

15.4.3.3.3 Creating a Schema from the Streaming Schema

Container

Create a new schema definition with initial properties.

Procedure

1. Right-click the Streaming Schema container and choose New Streaming Schema .

2. Complete the information in the General tab or other tabs.

You can complete schema definition properties at any time before generating the physical models.

3. Click OK to save the schema definition.

15.4.3.3.4 Generating Schema from an SAP IQ, SAP HANA, or SAP ASE Table

Follow the same steps as when generating a Streaming Schema model, selecting a single table to generate.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

277

15.4.3.4 Defining Schema Properties

Define schema details in the properties sheet.

Prerequisites

Add the schema definition to the Streaming Schema model.

Procedure

1. Open the Streaming Schema Properties sheet from the Browser tree or the diagram.

2. Edit fields on the General, Attributes, and Identifiers tabs.

3. (Optional) Right-click an attribute to open the Attribute Properties sheet.

4. (Optional) In the Attribute Properties sheet, choose More to see extended property details.

5. Click Apply to apply changes.

6. Click OK when done.

In this section:

General Tab Properties [page 279]

View information about the Name and Comment properties of a schema definition on the General tab of the Schema Definition Properties sheet.

Attributes Tab Properties [page 279]

The Attributes tab of the Schema Definition Properties sheet lets you quickly set information for all fields of a Streaming Schema.

Adding an Attribute to Schema [page 280]

Add fields to schema by adding attributes to the schema definition.

Identifiers [page 280]

An identifier is a column or combination of columns that uniquely defines a specific Streaming

Schema.

Defining Identifiers [page 281]

Define identifiers to indicate which schema attributes become keys in the destination table.

278

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

15.4.3.4.1 General Tab Properties

View information about the Name and Comment properties of a schema definition on the General tab of the

Schema Definition Properties sheet.

Property

Name

Description

Text that identifies the object's purpose for non-technical users, for example, Stock

Quote. This element is used for descriptive purposes only, and can contain any string.

Comment

An optional comment field. This is stored only in the model, not in the schema.

15.4.3.4.2 Attributes Tab Properties

The Attributes tab of the Schema Definition Properties sheet lets you quickly set information for all fields of a

Streaming Schema.

Property

Name

Description

Name of the field. The value can contain any string. This ele ment is used for descriptive purposes only.

Code

By default the code is generated from the name by applying the naming conventions specified in the model options. To decouple name-code synchronization, click to release the = button to the right of the Code field.

Smart data streaming Datatype

Select from the list of supported smart data streaming data types.

For information on smart data streaming datatypes, see Da tatypes [page 45] .

Data Type (Internal PowerDesigner datatype)

Length

Select from the list of supported datatypes.

For information on PowerDesigner datatypes, see related in formation in PowerDesigner>Data Modeling.

Required for money and decimal data.

Limited to precision 34. Precision must be the same on SAP

IQ and SAP ASE.

Not used for other datatypes.

Precision

Required for money and decimal data. Not used for other datatypes.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

279

Property

Domain

Description

Specifies a domain which defines the datatype and related data characteristics for the schema attribute. It may also in dicate check parameters, and business rules.

Select a domain from the list, or click the Ellipsis button to create a new domain in the List of Domains.

Attribute Properties Sheet

Each field in a schema definition has its own Properties sheet.

In the Attribute Properties Sheet, you can:

● View or edit the same information as in the Attributes tab of the Schema Definition Properties sheet

● Specify validation checks for an attribute

● View attribute dependencies

● View impact and lineage analyses for an attribute

15.4.3.4.3 Adding an Attribute to Schema

Add fields to schema by adding attributes to the schema definition.

Procedure

1. In the schema definition to which you are adding an attribute, do any of the following:

○ From the schema definition, right-click and choose New. This opens the Attribute Properties sheet.

○ From the Attributes tab in the Streaming SchemaProperties sheet, type information in the row below the last attribute.

○ From the Attributes tab in the Streaming Schema Properties sheet, click the Insert a Row, Add a Row, or Add Attributes or Replicate Attributes toolbar button from other schema definitions.

Before replicating attributes, read Object Replications in the PowerDesigner Core Features Guide.

2. Edit information in the Attributes Properties sheet or row as needed.

15.4.3.4.4 Identifiers

An identifier is a column or combination of columns that uniquely defines a specific Streaming Schema.

Identifiers in the Streaming Schema model become keys on tables in the SAP IQ, SAP HANA, and SAP ASE physical models.

280

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

Each Streaming Schema can have at most one primary identifier, which becomes the primary key in the generated table.

When an identifier has multiple attributes, the primary key in the destination table is composed of multiple columns. For example, in the sample model, the Dividend Event schema has one identifier. Attributes for this primary identifier are Instrument and Disbursed Date. Thus the primary key for the Dividend Event table is composed of both the Instrument and Disbursed Date columns.

15.4.3.4.5 Defining Identifiers

Define identifiers to indicate which schema attributes become keys in the destination table.

Procedure

1. Choose one of the following:

○ Right-click Streaming Schema and choose New Identifier .

○ (Primary identifiers only) On the Streaming Schema Properties sheet, select the Attributes tab, and click the Primary Identifier column (the narrow column with the header P) for each attribute that is part of the primary identifier. Skip the remaining steps.

Note

In the Streaming Schema Properties Attributes tab, a checkmark in the P column indicates a primary identifier.

2. Select the General tab in the Identifier Properties sheet: a. (Optional) Set the identifier name.

b. For a primary key, select Primary Identifier.

3. On the Attributes tab in the Identifier Properties sheet, enter the fields that identify the schema.

15.4.4 Validating a Model

Check the validity of your model after schema changes, and before generating schema templates, code, or a physical model. You can check the validity of a model at any time.

Procedure

1. (Optional) Select diagrams for the schema you want to validate.

2. Choose Tools Check Model (F4) .

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

281

3. In the Options tab of Check Model Parameters, expand the containers and choose validation checks.

The Options tab lists checks to be performed with symbols indicating their severity.

○ Do not disable any smart data streaming-specific checks.

○ (Default and recommended) Disable Existence of relationship or entity link under Entity.

4. In the Selection tab, navigate to the Streaming Schema subtab and select schema definitions to check:

○ Select or unselect check boxes.

○ Choose a named selection.

○ If you selected schema in your diagram before starting the model check, you can select them for checking by clicking Use Graphical Selection ( ) in the Selection tab toolbar.

5. Click OK.

Next Steps

Review results in the Check Model subtab in the status bar. It lists the checks made, and any errors or warnings.

Correct any errors. No automatic corrections are provided.

In this section:

PowerDesigner Validity Checks [page 282]

Standard PowerDesigner checks determine if a model is internally consistent and correct.

Custom Checks for Streaming Schema Extensions [page 283]

The Streaming Schema extension offers many custom checks.

15.4.4.1 PowerDesigner Validity Checks

Standard PowerDesigner checks determine if a model is internally consistent and correct.

For example:

● Each Streaming Schema name must be unique

● Each object name in the Streaming Schema model must be unique

● Each field must have an assigned smart data streaming datatype.

For descriptions of standard PowerDesigner checks, see Working with Data Models > Checking a Data Model in the PowerDesigner Data Modeling guide.

282

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

15.4.4.2 Custom Checks for Streaming Schema Extensions

The Streaming Schema extension offers many custom checks.

Checks for Each Schema

Custom checks under the Schema Definition type validate values in the General tab of the Schema Properties sheet.

Option

NameIsValid

Description

Validates that names of Streaming Schema are valid java identifiers.

Checks for Each Field in a Schema

Custom checks for fields are under the Entity Attribute type. They validate values in the Attributes tab of the

Streaming Schema Properties sheet.

Option

FieldNameIsValid

StreamingDatatypeExists

UniqueDestColumnName

Description

Validates that field names are valid java identifiers.

Validates that the datatype is specified.

Validates that DestColumnName is unique within that schema.

15.4.5 Importing a CCL File

Import the defined schema definitions in a CCL file into a Streaming Schema model.

Prerequisites

You have added the Everyone user to the PowerDesigner folder and have granted this user read and write permissions.

Procedure

1. Open the PowerDesigner model.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

283

2. In the Browser tree, right-click the Streaming Schema container and choose Import CCL File....

3. Navigate to the CCL file you wish to import.

4. Click OK to import the schema definitions defined in the CCL file.

Note

A warning message appears if the CCL file schema definitions are not valid. Resolve the errors before importing the CCL file. Navigate to the User/Appdata/Roaming/PowerDesigner/

StreamingCompiler/compiledOutput.log file to view the errors.

Results

The schema defined in the CCL file is imported into the Streaming Schema model.

15.4.6 Exporting a CCL File

Export all the defined schema from the Streaming Schema model into a CCL file for compiling and further analysis.

Procedure

1. Open the PowerDesigner model.

2. In the Browser tree, right-click the Streaming Schema container and choose Export CCL File....

3. Navigate to the CCL file you wish to export to.

4. Click OK to export the schema definitions.

Results

The schema defined in the Streaming Schema model is exported as a CCL file.

15.5 Model Generation

Model generation with the Streaming Schema models is a critical step in ensuring the integrity of your production environment.

To generate data models, you can perform one of the following tasks:

284

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

● Generate SAP IQ, HANA, and ASE physical data models from the Streaming Schema model.

● Generate a Streaming Schema logical data model from the SAP IQ, HANA, or ASE physical data model.

Each generation process relies on transformation rules for that model type, which are defined in the Streaming

Schema extensions for PowerDesigner.

In this section:

Generating a new SAP IQ, HANA, or ASE Model from a Streaming Schema Model [page 285]

Generate either an SAP IQ, HANA, or ASE physical data model from a Streaming Schema logical data model.

Generating a New Streaming Schema Model from an SAP IQ, HANA, or ASE Model [page 288]

Generate a new Streaming Schema logical data model from either SAP IQ, HANA, or ASE physical data models.

Updating an Existing SAP IQ, HANA, or ASE Model from a Streaming Schema Model [page 288]

Update either an SAP IQ, HANA, or ASE physical data model from a Streaming Schema logical data model.

Updating an Existing Streaming Schema Model from an SAP IQ, HANA, or ASE Model [page 289]

Update an existing Streaming Schema logical data model from either SAP IQ, HANA, or ASE physical data models.

15.5.1 Generating a new SAP IQ, HANA, or ASE Model from a

Streaming Schema Model

Generate either an SAP IQ, HANA, or ASE physical data model from a Streaming Schema logical data model.

Procedure

1. Open the Streaming Schema model.

2. From the PowerDesigner main menu, choose Tools Generate Physical Data Model .

3. In the General tab of the PDM Generation Options dialog, choose Generate new Physical Data Model.

4. For a new model, choose the target DBMS and the appropriate Name and Code.

○ For HANA, choose:

Field

DBMS

Name

Code

Value

SAP HANA Database 1.0

Keep the default, StreamingSchema, or enter another name.

Auto-generated from Name. For example, when Name is StreamingSchema, Code is StreamingSchema.

○ For ASE, choose:

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

285

Field

DBMS

Name

Code

Value

SAP Adaptive Server Enterprise 15.7

Keep the default, StreamingSchema_1 (the name of the container), or enter another name.

Auto-generated from Name. For example, when Name is StreamingSchema_1, Code is StreamingSchema_1.

○ For IQ, choose:

Field

DBMS

Name

Code

Value

SAP IQ 15.X

Note

Use latest version of SAP IQ available.

Keep the default, StreamingSchema_1 (the name of the container), or enter another name.

Auto-generated from Name. For example, when Name is StreamingSchema_1, Code is StreamingSchema_1.

5. Click the Detail tab.

6. (Optional) Choose Check model and Save generation dependencies.

7. Ensure that Enable transformations is selected.

8. Click the Extensions tab and ensure that the appropriate extension is selected:

○ ASE when generating a new SAP ASE model

○ IQ when generating a new SAP IQ model

○ HANA when generating a new SAP HANA model

9. On the Pre-generation and Post-generation tabs, ensure that all transformation rules are selected.

The post-generation tab appears only for new models.

10. On the Selection tab, select StreamingSchema to create tables for SAP IQ, SAP HANA, or SAP ASE, and choose OK.

Next Steps

After generation, check indexes, set physical options, and add foreign keys as needed.

In this section:

Checking Indexes [page 287]

PowerDesigner creates default indexes. Add, edit, or remove them as needed.

Setting Physical Options [page 287]

Set physical options for each table as needed for your SAP IQ or SAP ASE database.

Adding Foreign Keys [page 287]

Add foreign-key relationships to physical data models.

286

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

15.5.1.1 Checking Indexes

PowerDesigner creates default indexes. Add, edit, or remove them as needed.

Procedure

1. Open the new or updated physical data model.

2. For each table, right-click the table and choose Properties.

3. In the Indexes tab, edit indexes as needed for your data and performance requirements.

15.5.1.2 Setting Physical Options

Set physical options for each table as needed for your SAP IQ or SAP ASE database.

Procedure

1. Right-click the table and choose Properties.

2. Define any options needed:

○ (SAP ASE only) In the Physical Options (Common) tab, choose from the physical options most commonly set for the object.

○ In the Physical Options tab, choose from all available options.

○ (SAP ASE only) In the Partitions tab, set partitioning options for selected columns.

For more information on partitioning, see the SAP ASE and SAP IQ documentation sets.

15.5.1.3 Adding Foreign Keys

Add foreign-key relationships to physical data models.

Procedure

1. Add tables to the physical data model that are not in your Market Data diagram and that contain lookup columns for foreign keys.

New SAP ASE, SAP HANA, and SAP IQ models generated from a Streaming Schema model contain only market data tables.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

287

2. Right-click the table and choose Properties or Keys.

3. Add foreign-key relationships to tables that are not in the Market Data diagram.

15.5.2 Generating a New Streaming Schema Model from an

SAP IQ, HANA, or ASE Model

Generate a new Streaming Schema logical data model from either SAP IQ, HANA, or ASE physical data models.

Procedure

1. Open the IQ, HANA, or ASE model.

2. From the PowerDesigner main menu, choose Tools Generate Logical Data Model .

3. In the General tab of the LDM Generation Options dialog, choose Generate new Logical Data Model.

4. Specify a Name.

Code is autogenerated from the name.

5. On the Detail tab, choose Options:

○ (Optional) Check model

○ (Optional) Save generation dependencies

○ (Optional) Convert names into codes

○ (Required) Enable transformations

6. On the Extensions tab, choose StreamingSchema.

7. On the Selection tab, choose tables from which to generate schema.

8. Click OK.

15.5.3 Updating an Existing SAP IQ, HANA, or ASE Model from a Streaming Schema Model

Update either an SAP IQ, HANA, or ASE physical data model from a Streaming Schema logical data model.

Procedure

1. Open the Streaming Schema model.

2. From the PowerDesigner main menu, choose Tools Generate Physical Data Model .

3. In the General tab of the PDM Generation Options dialog, choose Update existing Physical Data Model.

288

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

4. Select the model and leave Preserve Modifications selected.

5. Click the Detail tab.

6. (Optional) Choose Check model and Save generation dependencies.

7. Ensure that Enable transformations is selected.

8. In the Merge Models dialog, confirm the updates you want, and click OK.

Next Steps

After generation, check indexes, set physical options, and add foreign keys as needed.

15.5.4 Updating an Existing Streaming Schema Model from an SAP IQ, HANA, or ASE Model

Update an existing Streaming Schema logical data model from either SAP IQ, HANA, or ASE physical data models.

Procedure

1. Open either the IQ, HANA, or ASE model.

2. From the PowerDesigner main menu, choose Tools Generate Logical Data Model .

3. In the General tab of the LDM Generation Options dialog, choose Update existing Logical Data Model.

4. Select the model and leave Preserve Modifications selected.

5. On the Detail tab, choose Options:

○ (Optional) Check model

○ (Optional) Save generation dependencies

○ (Optional) Convert names into codes

○ (Required) Enable transformations

6. On the Selection tab, choose tables from which to generate schema.

7. Click OK.

15.6 Impact and Lineage Analysis

With impact and lineage analysis, you can determine the full impact of changes to any object in the integrated model.

Impact analysis shows the effect of an action on the objects that depend on the initial object.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

289

Lineage analysis identifies the objects that influence the initial object.

You can perform these analyses on:

● A schema definition or any of its properties in the Streaming Schema logical data model

● A table or column in the SAP ASE, HANA, or IQ physical data model

The results shows the effect of a change throughout the logical and physical data models.

In this section:

Launching an Impact and Lineage Analysis [page 290]

Analyze the impact of a change to your model from the Impact and Lineage Analysis dialog box.

Generating an Analysis Diagram [page 292]

Generate an analysis diagram to view the impact or lineage analysis in graphical form.

Reviewing an Impact and Lineage Analysis [page 292]

Review the analysis in the preview or the impact and lineage model diagram.

Sample Analysis for a Schema Definition [page 293]

The sample analysis for a schema definition shows that the Bond History schema in the Streaming

Schema model was used to generate the BOND_HISTORY tables in the SAP HANA, ASE, and IQ models.

Sample Analysis for a Table [page 293]

The sample analysis for a table shows that the STOCK_QUOTE table was generated from the Stock

Quote schema definition in the Streaming Schema model.

15.6.1 Launching an Impact and Lineage Analysis

Analyze the impact of a change to your model from the Impact and Lineage Analysis dialog box.

Context

The Impact and Lineage Analysis dialog lets you review your analysis through:

● A preview – displays the impact and lineage analysis in a tree form. See PowerDesigner Core Features

Guide > Reviewing an Analysis in Preview.

● An impact analysis model (IAM) – displays the impact and lineage analysis in a diagram. See

PowerDesigner Core Features Guide > Reviewing an Analysis in an IAM Model.

Procedure

1. Open an impact and lineage analysis in any of these ways:

○ Select an object in the Browser or in the diagram and press Ctrl + F11.

○ Select one or more objects in the diagram and select Tools Impact and Lineage Analysis .

290

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

○ Right-click an object symbol in the diagram and select Edit Impact and Lineage Analysis .

○ Right-click an object entry in the Browser and select Impact and Lineage Analysis.

○ (When deleting an object) Click Impact on the Confirm Deletion dialog box.

○ Open an object's property sheet, click the Dependencies tab, then click Impact Analysis .

2. (Optional) Enter a name for your analysis result. This becomes the name of the generated model.

3. Select an impact rule set for your analysis. Choose one of these predefined rule sets:

○ Conceptual Impact Analysis – restrict the analysis to objects impacted by modeling changes on the initial object, such as a modification on a requirement definition.

○ Data Impact Analysis – identify the use, if any, of a value contained in the initial object.

○ Delete Impact Analysis – (default when deleting an object) restrict the analysis to objects that are directly impacted by the deletion of the initial object.

○ Global Impact Analysis – (default when not deleting an object) identify all the objects that depend on the initial object.

○ None – no impact rule set is selected.

4. Select a lineage rule set for your analysis. Choose one of these predefined rule sets:

○ Conceptual Lineage Analysis – justify the modeling existence of the initial object, and ensure it fulfills a well-identified need.

○ Data Lineage Analysis – identify the origin of the value contained in the initial object.

○ Global Lineage Analysis – (default when not deleting an object) identify all the objects that influence the initial object.

○ None – (default when deleting an object) no lineage rule set is selected.

5. (Optional) Click the Properties tool next to each rule set to review it. See PowerDesigner Core Features

Guide > Editing analysis rules.

Results

The analysis appears in the Impact and Lineage tab of the dialog box. See PowerDesigner Core Features Guide

> Reviewing an Analysis in Preview.

Note

You can click the Select Path tool to change the default folder for analysis rule sets, or click the List of Rule

Sets tool to open the List of Impact and Lineage Analysis Rule Sets window, and review a specific rule.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

291

15.6.2 Generating an Analysis Diagram

Generate an analysis diagram to view the impact or lineage analysis in graphical form.

Prerequisites

Launch an impact or lineage analysis.

Procedure

1. In the Impact and Lineage Analysis dialog, click Generate Diagram to view a graphical form of the analysis in its default diagram.

2. (Optional) Save (Ctrl+S) the diagram as an impact analysis model (IAM).

See PowerDesigner Core Features Guide > Reviewing an Analysis in an IAM Model.

15.6.3 Reviewing an Impact and Lineage Analysis

Review the analysis in the preview or the impact and lineage model diagram.

Procedure

1. Review the impact of the action and the lineage of the entity in the preview.

2. In the preview List tab, save the analysis in RTF or CSV format, or print.

3. You can refine your analysis by removing or adding initial objects, changing the analysis rule sets to be used, and customizing actions.

4. If you have generated an IAM, you can customize the display preferences and model options, print the model, and compare it with another IAM.

5. Watch for a red dot on an object icon in a generated model.

When you generate a model to another model or create an external shortcut, you create cross-model dependencies, which are taken into account during impact and lineage analysis.

When an object belonging to an unavailable related model is encountered, a red dot appears on the object icon and the analysis is interrupted. To continue, open the related model by right-clicking the object in the

IAM Browser or in the preview, and selecting Open Model.

292

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

15.6.4 Sample Analysis for a Schema Definition

The sample analysis for a schema definition shows that the Bond History schema in the Streaming Schema model was used to generate the BOND_HISTORY tables in the SAP HANA, ASE, and IQ models.

15.6.5 Sample Analysis for a Table

The sample analysis for a table shows that the STOCK_QUOTE table was generated from the Stock Quote schema definition in the Streaming Schema model.

Outgoing References shows foreign-key relationships. Streaming Schema definitions become Market Data diagram tables when generated to a PDM.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

293

Figure 6: Impact and Lineage Analysis for STOCK_QUOTE Table in ASE

15.7 DDL Script Generation

The corresponding adapter (SAP IQ, HANA, ASE) schema must match the SAP IQ, HANA, and ASE database schema for all tables in which data is inserted. After you make changes, use PowerDesigner to produce a set of data definition language (DDL) statements directly from the physical data model (SAP IQ, HANA, and ASE).

PowerDesigner saves the DDL statements in a SQL script to generate the tables and other objects for the target databases.

294

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

In this section:

Generating Database Schema with PowerDesigner [page 295]

PowerDesigner includes all of the necessary resources to generate a set of DDL statements in SQL scripts directly from the PowerDesigner data models. Run these scripts to generate a schema for your

SAP IQ, HANA, and ASE databases.

Generating DDL Scripts [page 296]

Generate DDL scripts directly from the SAP IQ, HANA, or ASE data model. PowerDesigner saves the results in a SQL script that you can use to generate the tables and other objects in the target database.

Executing DDL Scripts for the SAP IQ Database [page 298]

Execute the DDL script in Interactive SQL and create database objects in the SAP IQ database.

Executing DDL Scripts for the SAP HANA Database [page 298]

Execute the DDL script using hdbsql and create database objects in the SAP HANA database.

Executing DDL Scripts for the SAP ASE Database [page 299]

Execute the DDL script in Interactive SQL and create database objects in the SAP ASE database.

15.7.1 Generating Database Schema with PowerDesigner

PowerDesigner includes all of the necessary resources to generate a set of DDL statements in SQL scripts directly from the PowerDesigner data models. Run these scripts to generate a schema for your SAP IQ, HANA, and ASE databases.

Procedure

1. In PowerDesigner, open the data model.

2. Change the default database user.

3. Generate the script that creates a schema for the new database.

4. Log in to the database and run the script.

In this section:

Changing the Default Database User [page 296]

Overwrite the default database owner for the SAP IQ, HANA, or ASE database with a name specific to your environment.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

295

15.7.1.1 Changing the Default Database User

Overwrite the default database owner for the SAP IQ, HANA, or ASE database with a name specific to your environment.

Context

In the database, the user who creates an object (table, view, stored procedure, and so on) owns that object and is automatically granted all permissions on it. Overwriting the default user name globally changes ownership of database objects from the default owner to the new owner.

Procedure

1. Start PowerDesigner.

2. Select File Open and choose the database that you want to change the default owner of ( IQ.pdm,

HANA.pdm, or ASE.pdm).

3. Select Model Users and Roles Users .

4. In the Name and Code columns, change the default user to the new database user.

5. Click OK.

15.7.2 Generating DDL Scripts

Context

Use the model file of the database for which you wish to generate DDL scripts. For example, to generate DDL for the SAP ASE database, use the ASE model. When you have the model open, do not change the target database as doing so results in the loss of index information.

By default, the ASE.pdm data model includes only those indexes that support the sample queries. The statements that create these indexes are included in the DDL scripts, which means the indexes supplied with the model are created automatically when you run the corresponding DDL scripts.

You can add or remove indexes from the ASE data model. See the SAP ASE product documentation for details on SAP ASE indexes.

296

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

Procedure

1. Select Database Generate Database .

2. Browse to the directory where you want to store the script. Click OK.

3. Enter a name for the SQL script.

4. On the Options tab, verify that the options are set correctly:

Object

Domain

Table

Column

Key

Index

Options

Create User-Defined Data Type

Create Table

User Data Type

Create Primary Key Inside

○ Create Index

○ Index Filter Foreign Key

○ Index Filter Alternate Key

○ Index Filter Cluster

○ Index Filter Others

5. Click the Selection tab.

6. Choose the database owner.

7. On the Tables tab, click Select All.

8. On the Domains tab, choose the database owner, click Select All, click Apply, then click OK.

PowerDesigner checks the model for any errors, builds a result list, and generates the DDL. The Result dialog appears, which identifies the name and location of the generated file.

9. (Optional) Click Edit to view the generated script.

The Result List dialog appears in the background and may include several warnings, for example,

"Existence of index" and "Existence of reference". You can safely ignore these warnings.

10. Close the Result List dialog, then exit PowerDesigner.

11. PowerDesigner prompts:

○ To save the current workspace, click No.

○ To save the (modified) model, click Yes or No.

Results

● PowerDesigner prompts: To save the current workspace, click No.

● If PowerDesigner prompts you to save the model, click Yes to save the modified model. Otherwise, click

No.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

297

15.7.3 Executing DDL Scripts for the SAP IQ Database

Execute the DDL script in Interactive SQL and create database objects in the SAP IQ database.

Prerequisites

Start the SAP IQ database server if it is not running.

Procedure

1. In a command prompt, change to the directory that contains the database files and enter: start_iq -n <server-name> @ <config-file> .cfg <database-name> .db.

Use the -n switch to name the server, either in the configuration file or on the command line when you start the server.

Note

If you specify -n

<server-name>

without a

<database-name>

, you connect to the default database on the current server. If you specify -n

<database-name>

without a

<server-name>

, you connect to the specified database on the current server.

2. Enter dbisql.

3. Enter the correct user ID, password, and server information.

4. Open the generated DDL script for SAP IQ and click Execute SQL Statement on the toolbar.

15.7.4 Executing DDL Scripts for the SAP HANA Database

Execute the DDL script using hdbsql and create database objects in the SAP HANA database.

Prerequisites

Start the SAP HANA database server if it is not running.

298

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

Procedure

● In a command prompt, enter: hdbsql -n <host> : <port> -u <user> -p <password> -I <script-file>

15.7.5 Executing DDL Scripts for the SAP ASE Database

Execute the DDL script in Interactive SQL and create database objects in the SAP ASE database.

Prerequisites

Start the SAP ASE server if it is not running.

Procedure

1. In a command prompt, enter: isql -S <server-name> -U <user-name> -P <password> -i <ase-ddl.sql-file> -o

<log-file>

2. PowerDesigner prompts:

○ To save the current workspace. Click No

○ To save the (modified) model. Click Yes or No.

3. Check the log file for errors.

SAP HANA Smart Data Streaming: Developer Guide

SAP PowerDesigner for SAP HANA Smart Data Streaming

P U B L I C

299

16 Datatypes in SQL Queries

There are several methods of querying smart data streaming using SQL. All SQL statements follow these rules for dealing with the specified datatypes in SAP HANA smart data streaming.

Inserting Date and Data

To insert seconddate, msdate, and bigdatetime values into a SQL statement, use the unseconddate, untime, unmsdate, and unbigdatetime functions to convert the input values into the correct datatype: insert into win

(c_key,seconddate,time,msdate,bigdatetime) values(21478,unseconddate('2013-02-11

17:35:45'),untime(17:35:45'),unmsdate('2013-02-11 17:35:45'), unbigdatetime('2013-02-11 17:35:45'))

Specifying Money Data

When specifying the value for money data, append either

: insert into win

(c_key,c_money) values(111770,8.12345d)

There are 15 money datatypes with a fixed number of digits following the decimal point: money1 with one digit, money2 with two digits, ... money15 with 15 digits. When specifying a value for one of these datatypes, append either

and the number following money (for example

for money7): insert into win

(c_key,c_money1,c_money2,c_money3,c_money4,c_money5,c_money10,c_money15) values(111770,8.12345d1,8.12345d2,8.12345d3,8.12345d4,8.12345d5,8.123456789012345

d10,8.123456789012345d15)

Reading and Writing Binary Data

To write binary data to smart data streaming in a SQL statement, use the hex_binary() function to convert the string representation of the hexadecimal value to binary: insert into win

(c_key,c_binary) values(111770,hex_binary('9140ACA0361856334DD319F05'))

300

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Datatypes in SQL Queries

To read binary data from smart data streaming in an SQL statement, use the hex_string() function to convert the binary value to a string representation of the hexadecimal value: select c_key,hex_string(c_binary)from win

Related Information

Decimal and Money Datatypes [page 49]

SAP HANA Smart Data Streaming: Developer Guide

Datatypes in SQL Queries

P U B L I C

301

17 Appendix: List of Keywords

Reserved words in CCL that are case-insensitive. Keywords cannot be used as identifiers for any CCL objects.

age(s) auto continue default dynamic external for groups import into last log millisecond(s) new or partition records safedelete setRange sum top union while adapter attach connection deduced dumpfile exit flex group if row(s) set stream to typeof where insert language local millis name on parameter(s) rank sec start switch true upsert within join library memory minute(s) not out pattern return and break create delta end fby foreignJava having inherits is left max min nostart order partitions retain schema slack sync transaction update window all begin count delete else false foreign hash in as case day(s) desc eventCache filter from hour(s) inner keep like micros module nth outfile primary right second(s) static then type values xmlattributes asc cast declare distinct every first full hr input key load microsecond(s) money null output properties roundrobin select store(s) times typedef when xmlelement

302

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: List of Keywords

18 Appendix: Date and Time Programming

Set time zone parameters, date format code preferences, and define calendars.

In this section:

Time Zones [page 303]

A time zone is a geographic area that has adopted the same standard time, usually referred to as the local time.

Date and Time Format Codes [page 310]

The following lists valid components used to specify the format of date and time datatypes in SAP

HANA smart data streaming projects.

Calendar Files [page 315]

A text file detailing the holidays and weekends in a given time period.

18.1 Time Zones

A time zone is a geographic area that has adopted the same standard time, usually referred to as the local time.

Most adjacent time zones are one hour apart. By convention, all time zones compute their local time as an offset from GMT/UTC. GMT (Greenwich Mean Time) is a historical term, originally referring to mean solar time at the Royal Greenwich Observatory in the United Kingdom. GMT has been replaced by UTC (Coordinated

Universal Time), which is based on atomic clocks. For all SAP HANA smart data streaming purposes, GMT and

UTC are equivalent. Due to political and geographical practicalities, time zone characteristics may change over time. For example, the start date and end date of daylight saving time may change, or new time zones may be introduced in newly created countries.

Internally, SAP HANA smart data streaming always stores date and time type information as a number of seconds, milliseconds, or microseconds since midnight January 1, 1970 UTC (otherwise written as 1970-01-01

00:00:00 UTC), depending on the datatype. If a time zone designator is not used, UTC time is applied.

Daylight Saving Time

Daylight saving time is considered if the time zone uses daylight saving time and if the specified timestamp is in the time period covered by daylight saving time. The starting and ending dates for daylight saving time are stored in a C++ library.

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

P U B L I C

303

If the user specifies a particular time zone that uses daylight saving time, smart data streaming takes these dates into account to adjust the date and time datatype. For example, since Pacific Standard Time (PST) is in daylight saving time setting, the engine adjusts the timestamp accordingly: to_msdate('2002-06-18 13:52:00.123456 PST','YYYY-MM-DD HH24:MI:SS.ff TZD')

Transitioning from Standard Time to Daylight Saving Time and Vice-Versa

During the transition to and from daylight saving time, certain times do not exist. For example, in the U.S., during the transition from standard time to daylight saving time, the clock changes from 01:59 to 03:00; therefore 02:00 does not exist. Conversely, during the transition from daylight saving time to standard time,

01:00 to 01:59 appears twice during one night because the time changes from 2:00 to 1:00 when daylight saving time ends.

However, since there may be incoming data input during these undefined times, the engine must deal with them in some manner. During the transition to daylight saving time, smart data streaming interprets 02:59

PST as 01:59 PST. When transitioning back to standard time, smart data streaming interprets 02:00 PDT

(Pacific Daylight Time) as 01:00 PST.

In this section:

List of Time Zones [page 304]

SAP HANA smart data streaming supports standard time zones and their abbreviations.

18.1.1 List of Time Zones

SAP HANA smart data streaming supports standard time zones and their abbreviations.

This table lists the time zones used in SAP HANA smart data streaming from the industry-standard Olson time zone (also known as TZ) database.

ACT

ART

Africa/Accra

Africa/Asmera

Africa/Banjul

Africa/Brazzaville

Africa/Casablanca

Africa/Dakar

Africa/Douala

Africa/Gaborone

Africa/Kampala

AET

AST

Africa/Addis_Ababa

Africa/Bamako

Africa/Bissau

Africa/Bujumbura

Africa/Ceuta

Africa/Dar_es_Salaam

Africa/El_Aaiun

Africa/Harare

Africa/Khartoum

AGT

Africa/Abidjan

Africa/Algiers

Africa/Bangui

Africa/Blantyre

Africa/Cairo

Africa/Conakry

Africa/Djibouti

Africa/Freetown

Africa/Johannesburg

Africa/Kigali

304

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

Africa/Kinshasa

Africa/Lome

Africa/Lusaka

Africa/Maseru

Africa/Monrovia

Africa/Niamey

Africa/Porto-Novo

Africa/Tripoli

America/Adak

America/Antigua

Africa/Lagos

Africa/Luanda

Africa/Malabo

Africa/Mbabane

Africa/Nairobi

Africa/Nouakchott

Africa/Sao_Tome

Africa/Tunis

America/Anchorage

America/Araguaina

America/Argentina/Catamarca

America/Argentina/Jujuy

America/Argentina/ComodRivadavia

America/Argentina/La_Rioja

America/Argentina/Rio_Gallegos America/Argentina/San_Juan

America/Argentina/Ushuaia America/Aruba

America/Atka

America/Belem

America/Bogota

America/Cambridge_Bay

America/Bahia

America/Belize

America/Boise

America/Campo_Grande

America/Caracas

America/Cayman

America/Coral_Harbour

America/Cuiaba

America/Dawson

America/Catamarca

America/Chicago

America/Cordoba

America/Curacao

America/Dawson_Creek

America/Detroit

America/Eirunepe

America/Fort_Wayne

America/Godthab

America/Grenada

America/Guayaquil

America/Havana

America/Indiana/Knox

America/Indiana/Vevay

America/Inuvik

America/Jujuy

America/Kentucky/Monticello

America/Lima

America/Dominica

America/El_Salvador

America/Fortaleza

America/Goose_Bay

America/Guadeloupe

America/Guyana

America/Hermosillo

America/Indiana/Marengo

America/Indiana/Vincennes

America/Iqaluit

America/Juneau

America/Knox_IN

America/Los_Angeles

Africa/Libreville

Africa/Lubumbashi

Africa/Maputo

Africa/Mogadishu

Africa/Ndjamena

Africa/Ouagadougou

Africa/Timbuktu

Africa/Windhoek

America/Anguilla

America/Argentina/Buenos_Aires

America/Argentina/Cordoba

America/Argentina/Mendoza

America/Argentina/Tucuman

America/Asuncion

America/Barbados

America/Boa_Vista

America/Buenos_Aires

America/Cancun

America/Cayenne

America/Chihuahua

America/Costa_Rica

America/Danmarkshavn

America/Denver

America/Edmonton

America/Ensenada

America/Glace_Bay

America/Grand_Turk

America/Guatemala

America/Halifax

America/Indiana/Indianapolis

America/Indiana/Petersburg

America/Indianapolis

America/Jamaica

America/Kentucky/Louisville

America/La_Paz

America/Louisville

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

P U B L I C

305

America/Maceio

America/Martinique

America/Menominee

America/Miquelon

America/Montevideo

America/Nassau

America/Nome

America/Panama

America/Phoenix

America/Porto_Acre

America/Rainy_River

America/Regina

America/Santiago

America/Scoresbysund

America/St_Kitts

America/St_Vincent

America/Thule

America/Toronto

America/Virgin

America/Yakutat

Antarctica/Davis

Antarctica/McMurdo

Antarctica/South_Pole

Arctic/Longyearbyen

Asia/Amman

Asia/Aqtobe

Asia/Baghdad

Asia/Bangkok

Asia/Brunei

Asia/Chongqing

Asia/Dacca

Asia/Dili

Asia/Gaza

Asia/Hovd

Asia/Jakarta

Asia/Kabul

Asia/Aden

Asia/Anadyr

Asia/Ashgabat

Asia/Bahrain

Asia/Beirut

Asia/Calcutta

Asia/Chungking

Asia/Damascus

Asia/Dubai

Asia/Harbin

Asia/Irkutsk

Asia/Jayapura

Asia/Kamchatka

America/Managua

America/Mazatlan

America/Merida

America/Moncton

America/Montreal

America/New_York

America/Noronha

America/Pangnirtung

America/Port-au-Prince

America/Porto_Velho

America/Rankin_Inlet

America/Rio_Branco

America/Santo_Domingo

America/Shiprock

America/St_Lucia

America/Swift_Current

America/Thunder_Bay

America/Tortola

America/Whitehorse

America/Yellowknife

Antarctica/DumontDUrville

Antarctica/Palmer

Antarctica/Syowa

306

P U B L I C

Asia/Almaty

Asia/Aqtau

Asia/Ashkhabad

Asia/Baku

Asia/Bishkek

Asia/Choibalsan

Asia/Colombo

Asia/Dhaka

Asia/Dushanbe

Asia/Hong_Kong

Asia/Istanbul

Asia/Jerusalem

Asia/Karachi

America/Manaus

America/Mendoza

America/Mexico_City

America/Monterrey

America/Montserrat

America/Nipigon

America/North_Dakota/Center

America/Paramaribo

America/Port_of_Spain

America/Puerto_Rico

America/Recife

America/Rosario

America/Sao_Paulo

America/St_Johns

America/St_Thomas

America/Tegucigalpa

America/Tijuana

America/Vancouver

America/Winnipeg

Antarctica/Casey

Antarctica/Mawson

Antarctica/Rothera

Antarctica/Vostok

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

Asia/Kashgar

Asia/Kuala_Lumpur

Asia/Macao

Asia/Makassar

Asia/Nicosia

Asia/Oral

Asia/Pyongyang

Asia/Rangoon

Asia/Riyadh88

Asia/Sakhalin

Asia/Shanghai

Asia/Tashkent

Asia/Tel_Aviv

Asia/Tokyo

Asia/Ulan_Bator

Asia/Vladivostok

Asia/Yerevan

Atlantic/Canary

Atlantic/Jan_Mayen

Atlantic/South_Georgia

Australia/ACT

Australia/Broken_Hill

Australia/Darwin

Australia/Lindeman

Australia/NSW

Australia/Queensland

Australia/Tasmania

Australia/Yancowinna

Brazil/Acre

Brazil/West

CNT

CTT

Canada/East-Saskatchewan

Canada/Newfoundland

Canada/Yukon

Cuba

Asia/Katmandu

Asia/Kuching

Asia/Macau

Asia/Manila

Asia/Novosibirsk

Asia/Phnom_Penh

Asia/Qatar

Asia/Riyadh

Asia/Riyadh89

Asia/Samarkand

Asia/Singapore

Asia/Tbilisi

Asia/Thimbu

Asia/Ujung_Pandang

Asia/Urumqi

Asia/Yakutsk

Atlantic/Azores

Atlantic/Cape_Verde

Atlantic/Madeira

Atlantic/St_Helena

Australia/Adelaide

Australia/Canberra

Australia/Hobart

Australia/Lord_Howe

Australia/North

Australia/South

Australia/Victoria

BET

Brazil/DeNoronha

CAT

CST

Canada/Atlantic

Canada/Eastern

Canada/Pacific

Chile/Continental

EAT

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

Asia/Krasnoyarsk

Asia/Kuwait

Asia/Magadan

Asia/Muscat

Asia/Omsk

Asia/Pontianak

Asia/Qyzylorda

Asia/Riyadh87

Asia/Saigon

Asia/Seoul

Asia/Taipei

Asia/Tehran

Asia/Thimphu

Asia/Ulaanbaatar

Asia/Vientiane

Asia/Yekaterinburg

Atlantic/Bermuda

Atlantic/Faeroe

Atlantic/Reykjavik

Atlantic/Stanley

Australia/Brisbane

Australia/Currie

Australia/LHI

Australia/Melbourne

Australia/Perth

Australia/Sydney

Australia/West

BST

Brazil/East

CET

CST6CDT

Canada/Central

Canada/Mountain

Canada/Saskatchewan

Chile/EasterIsland

ECT

P U B L I C

307

EET

Egypt

Etc/GMT+0

Etc/GMT+11

Etc/GMT+3

Etc/GMT+6

Etc/GMT+0

Etc/GMT-10

Etc/GMT-13

Etc/GMT-3

Etc/GMT-6

Etc/GMT-9

Etc/UCT

Etc/Zulu

Europe/Athens

Europe/Berlin

Europe/Bucharest

Europe/Copenhagen

Europe/Helsinki

Europe/Kiev

Europe/London

Europe/Malta

Europe/Monaco

Europe/Oslo

Europe/Riga

Europe/San_Marino

Europe/Skopje

Europe/Tallinn

Europe/Uzhgorod

Europe/Vienna

Europe/Zagreb

Factory

GMT

GMT0

Hongkong

Iceland

EST

Eire

Etc/GMT+1

Etc/GMT+12

Etc/GMT+4

Etc/GMT+7

Etc/GMT-0

Etc/GMT-11

Etc/GMT-14

Etc/GMT-4

Etc/GMT-7

Etc/GMT0

Etc/UTC

Europe/Amsterdam

Europe/Belfast

Europe/Bratislava

Europe/Budapest

Europe/Dublin

Europe/Istanbul

Europe/Lisbon

Europe/Luxembourg

Europe/Mariehamn

Europe/Moscow

Europe/Paris

Europe/Rome

Europe/Sarajevo

Europe/Sofia

Europe/Tirane

Europe/Vaduz

Europe/Vilnius

Europe/Zaporozhye

GMT+0

Greenwich

IET

Indian/Antananarivo

308

P U B L I C

EST5EDT

Etc/GMT

Etc/GMT+10

Etc/GMT+2

Etc/GMT+5

Etc/GMT+8

Etc/GMT-1

Etc/GMT-12

Etc/GMT-2

Etc/GMT-5

Etc/GMT-8

Etc/Greenwich

Etc/Universal

Europe/Andorra

Europe/Belgrade

Europe/Brussels

Europe/Chisinau

Europe/Gibraltar

Europe/Kaliningrad

Europe/Ljubljana

Europe/Madrid

Europe/Minsk

Europe/Nicosia

Europe/Prague

Europe/Samara

Europe/Simferopol

Europe/Stockholm

Europe/Tiraspol

Europe/Vatican

Europe/Warsaw

Europe/Zurich

GB-Eire

GMT-0

HST

IST

Indian/Chagos

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

Indian/Christmas

Indian/Kerguelen

Indian/Mauritius

Iran

Jamaica

Libya

MST

Mexico/BajaSur

Mideast/Riyadh88

NST

Navajo

PRC

PST8PDT

Pacific/Chatham

Pacific/Enderbury

Pacific/Funafuti

Pacific/Guadalcanal

Pacific/Johnston

Pacific/Kwajalein

Pacific/Midway

Pacific/Norfolk

Pacific/Palau

Pacific/Port_Moresby

Pacific/Samoa

Pacific/Tongatapu

Pacific/Wallis

Portugal

SST

SystemV/AST4ADT

SystemV/EST5

SystemV/MST7

SystemV/PST8PDT

Turkey

US/Aleutian

US/East-Indiana

US/Indiana-Starke

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

Indian/Cocos

Indian/Mahe

Indian/Mayotte

Israel

Japan

MET

MST7MDT

Mexico/General

Mideast/Riyadh89

PLT

PRT

Pacific/Apia

Pacific/Easter

Pacific/Fakaofo

Pacific/Galapagos

Pacific/Guam

Pacific/Kiritimati

Pacific/Majuro

Pacific/Nauru

Pacific/Noumea

Pacific/Pitcairn

Pacific/Rarotonga

Pacific/Tahiti

Pacific/Truk

Pacific/Yap

ROC

Singapore

SystemV/CST6

SystemV/EST5EDT

SystemV/MST7MDT

SystemV/YST9

UCT

US/Arizona

US/Eastern

US/Michigan

P U B L I C

309

Indian/Comoro

Indian/Maldives

Indian/Reunion

JST

Kwajalein

MIT

Mexico/BajaNorte

Mideast/Riyadh87

NET

NZ-CHAT

PNT

PST

Pacific/Auckland

Pacific/Efate

Pacific/Fiji

Pacific/Gambier

Pacific/Honolulu

Pacific/Kosrae

Pacific/Marquesas

Pacific/Niue

Pacific/Pago_Pago

Pacific/Ponape

Pacific/Saipan

Pacific/Tarawa

Pacific/Wake

Poland

ROK

SystemV/AST4

SystemV/CST6CDT

SystemV/HST10

SystemV/PST8

SystemV/YST9YDT

US/Alaska

US/Central

US/Hawaii

US/Mountain

US/Pacific

UTC

W-SU

US/Pacific-New

Universal

WET

US/Samoa

VST

Zulu

18.2 Date and Time Format Codes

The following lists valid components used to specify the format of date and time datatypes in SAP HANA smart data streaming projects.

SAP HANA smart data streaming uses three separate classes of format codes for formatting and parsing date/time information: SimpleDateFormat, strftime(), and Time Formatting. Certain formats apply for specific components of smart data streaming:

● External toolkit adapters use SimpleDateFormat codes.

● Internal adapters use strftime() codes.

● CCL functions use SAP HANA smart data streaming Time Formatting codes and strftime() codes.

● Toolkit utilities use data formatting parameters that use strftime() codes.

SimpleDateFormat Codes

W y

Letter

External toolkit adapters are Java-based, and use SimpleDateFormat for formatting and parsing date/time values. The following rules apply:

● If the code is being specified inside a CCL file, and the value contains a quote within it, escape the quote with a backslash. For example, enter xmllistSecondDateFormat = 'yyyy-MM-dd'T'HH:mm:ss' as

'yyyy-MM-dd\'T\'HH:mm:ss'.

● Date and time formats are specified by date and time pattern strings. Within date and time pattern strings, unquoted letters from 'A' to 'Z' and from 'a' to 'z' are interpreted as pattern letters representing the components of a date/time string. Text can be quoted using single quotes (') to avoid interpretation. A single quote is represented by double-quotes around single quotes ("''"). All other characters are not interpreted; they're simply copied into the output string during formatting or matched against the input string during parsing.

The following pattern letters are defined (all other characters from 'A' to 'Z' and from 'a' to 'z' are reserved):

Date/Time Component

Era designator

Year

Month in year

Week in year

Week in month

Presentation

Text

Year

Number

Examples

1996; 96

July; Jul; 07

310

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

H k

K h d

E a

Letter

S z

Z m s

Date/Time Component

Day in year

Day in month

Day of week in month

Day in week

Am/pm marker

Hour in day (0-23)

Hour in day (1-24)

Hour in am/pm (0-11)

Hour in am/pm (1-12)

Minute in hour

Second in minute

Millisecond

Time zone

Presentation

Number

Text

Number

General time zone

RFC 822 time zone

Examples

189

Tuesday; Tue

978

Time; PST; GMT-8:00

-0800

Strftime() MsDate Conversion Codes

For internal adapters, CCL functions, and command line utilities, specify date/time formats using a subset of the C++ strftime() function codes. The following rules apply:

● Any date/time format specification that includes a percent sign (%) is considered a strftime() code.

● Some strftime() codes are valid only on Windows, or only on UNIX-like operating systems. Different implementations of strftime() also include minor differences in code interpretation. To avoid errors, ensure that both the server and the SAP HANA studio are on the same platform, and are using compatible strftime() implementations. It is also essential to confirm that the provided codes meet the requirements for the platform.

● All time zones for formats specified with strftime() are assumed to be the UTC time zone.

Smart data streaming supports the following strftime() codes:

Strftime() Code

Description

Abbreviated weekday name; for example "Mon".

Full weekday name; for example "Monday".

Abbreviated month name; for example "Feb".

Full month name; for example "February".

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

P U B L I C

311

Strftime() Code

312

P U B L I C

Description

Full date and time string: the output format for this code differs depending on whether you are using Windows or a

UNIX-like operating system.

● Windows: 08/26/08 20:00:00

● UNIX-like operating system: Tue Aug 26 20:00:00

2008

Day of the month, represented as a two-digit decimal inte ger with a value between 01 and 31.

Hour, represented as a two-digit decimal integer with a value between 00 and 23.

Hour, represented as a two-digit decimal integer with a value between 01 and 12.

Day of the year, represented as a three-digit decimal inte ger with a value between 001 and 366.

Month, represented as a two-digit decimal integer with a value between 01 and 12.

Minute, represented as a two-digit decimal integer with a value between 00 and 59.

Locale's equivalent of AM or PM.

Second, represented as a two-digit decimal integer with a value between 00 and 61.

Number of the week in the year, represented as a two-digit decimal integer with a value between 00 and 53, with Sun day considered the first day of the week.

Weekday number, represented as a one-digit decimal inte ger with a value between 0 and 6, with Sunday represented as 0.

Number of the week in the year, represented as a two-digit decimal integer with a value between 00 and 53, with Mon day considered the first day of the week.

Full date string (no time): The output format for this code differs, depending on whether you are using Windows or a

UNIX-like operating system.

● Windows: 08/26/08

● UNIX-like operating system output example: Tue Aug

26 2008

Full time string (no date).

Year, without the century, represented as a two-digit deci mal number with a value between 00 and 99.

Year, with the century, represented as a four-digit decimal number.

Replaced by %.

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

Smart Data Streaming Time Formatting Codes

DDD

DAY

HH12

HH24

SSSSS

Column Code

YYYY

YYY

MON

MONTH

For CCL functions, date and time formats are primarily specified using smart data streaming format codes.

Note

All designations of year, month, day, hour, minute, or second can also read a fewer number of digits than is specified by the code. For example, DD reads both two-digit and one-digit day entries.

Description

Month (01-12; JAN = 01).

Four-digit year.

Last three digits of year.

Last two digits of year.

Last digit of year.

Quarter of year (1, 2, 3, 4; JAN-MAR = 1).

Abbreviated name of month (JAN, FEB, ..., DEC).

Name of month, padded with blanks to nine characters (JANU

ARY, FEBRUARY, ..., DECEMBER).

Roman numeral month (1-XII; JAN = I).

Week of year (1-53), where week 1 starts on the first day of the year and continues to the seventh day of the year.

Week of month (1-5), where week 1 starts on the first day of the month and continues to the seventh day of the month.

Day of week (1-7; SUNDAY = 1).

Input

Day of month (1-31).

Day of year (1-366).

Name of day (SUNDAY, MONDAY, ..., SATURDAY).

Abbreviated name of day (SUN, MON, ..., SAT).

Hour of day (1-12).

Hour of day (0-23).

Meridian indicator (AM/PM).

Minute (0-59).

Second (0-59).

Seconds past midnight (0-86399).

Seconds since epoch (January 1, 1970 UTC). This format can only be used by itself, with the FF format, and/or with the time zone codes TZD, TZR, TZH and TZM.

Output

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

P U B L I C

313

TZD

TZH

TZM

RRRR

Column Code

MIC

FF[1-9]

Description

Microseconds since epoch (January 1, 1970 UTC).

Fractions of seconds (0-999999). When used in output, FF produces six digits for microseconds. FFFF produces 12 digits, repeating the six digits for microseconds twice. (In most cir cumstances, this is not the desired effect.) When used in input,

FF collects all digits until a non-digit is detected, and then uses only the first six, discarding the rest.

Fractions of seconds. For output only, produces the specified number of digits, rounding or padding with trailing zeros as needed.

Input

Milliseconds since epoch (January 1, 1970 UTC). When used for input, this format code can only be combined with FF (mi croseconds) and the time zone codes TZD, TZR, TZH, TZM. All other format code combinations generate errors. In addition, when MS is used with FF, the MS code must precede the FF code: for example, MS.FF.

Fill mode toggle: suppress zeros and blanks or not (default: not).

Y Exact mode toggle: match case and punctuations exactly (de fault: not).

Lets you store 20th century dates in the 21st century using only two digits.

Round year. Accepts either four-digit or two-digit input. If twodigit, provides the same return as RR.

Abbreviated time zone designator such as PST.

Time zone hour displacement. For example, -5 indicates a time zone five hours earlier than GMT.

N Time zone hour and minute displacement. For example, -5:30 indicates a time zone that is five hours and 30 minutes earlier than GMT.

Time zone region name. For example, US/Pacific for PST.

Output

TZR Y

Note

Format CCL functions using smart data streaming format unless this produces incorrect results. If this occurs, use the strftime() format.

314

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

18.3 Calendar Files

A text file detailing the holidays and weekends in a given time period.

Syntax

weekendStart <integer> weekendEnd <integer> holiday yyyy-mm-dd holiday yyyy-mm-dd

...

Components

Component weekendStart weekendEnd holiday

Description

An integer that represents a day of the week, when Mon day=0, Tuesday=1, ..., Saturday=5, and Sunday=6.

A day of the year, in the form yyyy-mm-dd. A calendar file can have unlimited holidays.

Usage

A calendar file is a text file that describes the start and end date of a weekend, and the holidays within the year.

Lines beginning with '#' characters are ignored, and may be used to provide user clarification or comments.

Calendar files are loaded and cached on demand. If changes occur in any of the calendar files, the refresh_calendars command must be sent to refresh the cached calendar data.

Example

The following is an example of a legal calendar file:

# SAP calendar data for US 2015 weekendStart 5 weekendEnd 6 holiday 2015-01-01 holiday 2015-05-25 holiday 2015-07-03

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

P U B L I C

315

holiday 2015-09-07 holiday 2015-11-26 holiday 2015-11-27 holiday 2015-12-24 holiday 2015-12-25

316

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Date and Time Programming

19 Appendix: Tutorial for Building and

Testing a Project in Studio

Walk through this hands-on tutorial to create, compile, and test a simple project in the studio. This is a great way of getting started in and familiar with studio and SAP HANA smart data streaming.

Context

The portfolio valuation sample project demonstrates how you can easily define streams and windows by attaching a previously configured adapter and discovering its schema, or by manually defining a window and its schema. It shows you how to define continuous queries—aggregations, joins, and more—using the visual tools in the SAP HANA Streaming Development perspective.

After you build the project, the tutorial shows you how to compile, run, and test the project you have created using tools in the SAP HANA Streaming Run-Test perspective.

In this section:

About the Portfolio Valuation Sample Project [page 320]

The portfolio valuation project that you build in this tutorial applies current prices to a portfolio of investments to compute the value of each investment and of the portfolio. It uses simple queries to aggregate and join data from two input windows, then outputs the refined data into SAP HANA using an output adapter.

Editing a Project Diagram [page 325]

In studio, define a new set of processing instructions for event data.

Edit projects in the visual editor by adding shapes from the palette to the project diagram, connecting them, and completing the configuration of each shape.

Discovering a Schema [page 327]

Attach an adapter by inserting it in the diagram, connecting it to a stream or window, and setting properties.

In studio, use the Schema Discovery button in the adapter shape to discover and automatically create a schema based on the format of the data from the adapter.

Adding an Input Window Manually [page 329]

In studio, add an input window to the diagram in the sample project

my_portfolio_valuation

Creating an Aggregate as a Simple Query [page 331]

In studio, add an aggregate simple query to the sample diagram to create a volume weighted average price (VWAP).

Creating a Join as a Simple Query [page 333]

Add a join to the sample project. A join combines events from two or more inputs to create a single stream or window. It is similar to a join in SQL.

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

317

Preparing Data for Output [page 337]

Add a new aggregate to finalize the data and prepare it for output into SAP HANA.

Preparing to Compile [page 339]

Create an SAP HANA service definition, and insert and configure an SAP HANA Output adapter using the studio.

Clean up the diagram by removing unused elements.

Compiling the Project and Viewing Problems [page 341]

Compile a project in studio before running it to check for errors and make corrections. Use the

Problems view to view error details.

Deploying the Sample Project [page 342]

Run the project and watch it open in the SAP HANA Streaming Run-Test perspective.

Loading Data into the Sample Project [page 343]

Test the sample project by loading reference data into the Positions window.

Play back the previously recorded price feed data, and view the continuous portfolio valuations in the sample project.

Additional Sample Projects [page 349]

This variation of the portfolio valuation project provides a complex, real-world example. The project uses a defined module with a named schema to easily scale out the application in a very high volume deployment. The SAP HANA Output adapter is not included in this module.

Smart data streaming in SAP HANA studio includes several completed example projects.

About the Portfolio Valuation Sample Project [page 320]

Editing a Project Diagram [page 325]

In studio, define a new set of processing instructions for event data.

Edit projects in the visual editor by adding shapes from the palette to the project diagram, connecting them, and completing the configuration of each shape.

Discovering a Schema [page 327]

Attach an adapter by inserting it in the diagram, connecting it to a stream or window, and setting properties.

In studio, use the Schema Discovery button in the adapter shape to discover and automatically create a schema based on the format of the data from the adapter.

Adding an Input Window Manually [page 329]

In studio, add an input window to the diagram in the sample project

my_portfolio_valuation

Creating an Aggregate as a Simple Query [page 331]

In studio, add an aggregate simple query to the sample diagram to create a volume weighted average price (VWAP).

318

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

Creating a Join as a Simple Query [page 333]

Add a join to the sample project. A join combines events from two or more inputs to create a single stream or window. It is similar to a join in SQL.

Preparing Data for Output [page 337]

Add a new aggregate to finalize the data and prepare it for output into SAP HANA.

10.

Preparing to Compile [page 339]

Create an SAP HANA service definition, and insert and configure an SAP HANA Output adapter using the studio.

11.

Clean up the diagram by removing unused elements.

12.

Compiling the Project and Viewing Problems [page 341]

Compile a project in studio before running it to check for errors and make corrections. Use the

Problems view to view error details.

13.

Deploying the Sample Project [page 342]

Run the project and watch it open in the SAP HANA Streaming Run-Test perspective.

14.

Loading Data into the Sample Project [page 343]

Test the sample project by loading reference data into the Positions window.

15.

Play back the previously recorded price feed data, and view the continuous portfolio valuations in the sample project.

16.

Additional Sample Projects [page 349]

17.

Smart data streaming in SAP HANA studio includes several completed example projects.

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

319

19.1 About the Portfolio Valuation Sample Project

Figure 7: Portfolio Valuation Sample Diagram (Iconic Mode)

The portfolio valuation project consists of various elements, any of which you can create either in the visual editor, or by manually writing CCL code.

CCL Element

Receives a stream of prices in an input window called PriceFeed. The schema for this window has five columns: Id, Symbol, Price, Shares, and TradeTime. The window uses the Id field as a primary key, and is set to keep the last 10 minutes of price events:

Diagram Element

CREATE INPUT WINDOW PriceFeed

SCHEMA

( Id integer ,

Symbol string ,

TradeTime seconddate ,

Price float ,

Shares integer )

PRIMARY KEY ( Id )

KEEP 10 MIN ;

320

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

CCL Element

Applies an aggregate simple query to create a 10-minute moving average—a volume weighted average price (VWAP). With the VWAP, you can see the value of positions based on the average price, rather than see the value of your positions change with every small price movement. The VWAP formula is calculated as:

Diagram Element

/**@SIMPLEQUERY=AGGREGATE*/

CREATE OUTPUT WINDOW VWAP

PRIMARY KEY DEDUCED

KEEP ALL ROWS

SELECT

PriceFeed.Symbol Symbol ,

PriceFeed.TradeTime LastTime ,

PriceFeed.Price LastPrice ,

sum ( PriceFeed.Price *

PriceFeed.Shares ) /

sum ( PriceFeed.Shares ) VWAP

FROM PriceFeed

GROUP BY PriceFeed.Symbol ;

Reads data from another input window, Positions, with three col umns: BookId, Symbol, and SharesHeld:

CREATE INPUT WINDOW Positions

SCHEMA

( BookId string ,

Symbol string ,

SharesHeld integer )

PRIMARY KEY ( BookId, Symbol) KEEP ALL

ROWS ;

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

321

CCL Element

Applies a join simple query, joining the market price (from the VWAP aggregate) to your holdings (Positions), so that you can see the value of your position in each stock:

Diagram Element

/**@SIMPLEQUERY=JOIN*/

CREATE OUTPUT WINDOW IndividualPositions

PRIMARY KEY DEDUCED

KEEP ALL ROWS

SELECT

VWAP.LastTime LastTime ,

VWAP.LastPrice LastPrice ,

VWAP.VWAP VWAP ,

Positions.BookId BookId ,

Positions.Symbol Symbol ,

Positions.SharesHeld SharesHeld ,

VWAP.LastPrice *

Positions.SharesHeld CurrentPosition ,

VWAP.VWAP *

Positions.SharesHeld AveragePosition

FROM VWAP RIGHT JOIN Positions ON

VWAP.Symbol = Positions.Symbol ;

Applies one more aggregation to show the total value of each

"book." This aggregate, ValueByBook, groups current and average values for individual positions into different "books." Each book may comprise a set of investment portfolios or funds. In the CCL, a

GROUP BY clause performs the aggregation:

/**@SIMPLEQUERY=AGGREGATE*/

CREATE OUTPUT WINDOW ValueByBook

PRIMARY KEY DEDUCED

KEEP ALL ROWS

SELECT

IndividualPositions.BookId BookId ,

sum

( IndividualPositions.CurrentPosition )

CurrentPosition ,

sum

( IndividualPositions.AveragePosition )

AveragePosition

FROM IndividualPositions

GROUP BY IndividualPositions.BookId ;

ATTACH INPUT ADAPTER Adapter1

TYPE toolkit_file_xmllist_input

Positions

PROPERTIES dir =

' <SAP HANA-studio-workspace> \exampledata', file = 'positions.xml' ;

322

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

CCL Element

Attaches an output adapter to output the processed data to an SAP

HANA table. This adapter requires an existing connection to SAP

HANA, which is created using the SAP HANA service from the Data

Services view. See Adding a Connection to an SAP HANA Database in the SAP HANA Smart Data Streaming: Configuration and Administra

tion Guide for detailed instructions on creating the service.

Diagram Element

ATTACH OUTPUT ADAPTER toSAPHANA TYPE hana_out

TO ValueByBook

PROPERTIES service = 'hanaservice' , table = 'exampletable' , dataWarehouseMode = 'ON' , msdateColumnName = 'TIME' ;

Column Alias in Expressions

Each expression defines a unique name or alias for the column.

In the portfolio valuation sample project, a derived window called VWAP takes input from an input stream

(PriceFeed) with columns Symbol, Price and TradeTime, and it includes an aggregate expression. Columns aliases for this derived window (created in visual editor as an aggregate simple query) are the following:

Alias

Symbol

LastPrice

VWAP

LastTime

Column Expression

PriceFeed.Symbol

PriceFeed.Price

( sum ( ( PriceFeed.Price * CAST ( FLOAT , Price

Feed.Shares ) ) ) / CAST ( float , sum ( Price

Feed.Shares ) ) )

PriceFeed.TradeTime

Parent topic: Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Next task: Creating the Project [page 324]

Related Information

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

323

19.2 Creating the Project

In studio, define a new set of processing instructions for event data.

Prerequisites

Start SAP HANA studio.

Procedure

1. Select File New Project... , then select SAP HANA smart data streaming New streaming

Project ,

2. For the purposes of this tutorial, in the Name field, enter

my_portfolio_valuation

3. In the Directory field, accept the default location or browse to a directory in which to store the new project folder.

Smart data streaming creates three files in the named directory:

<project-name> .ccl

<project-name> .cclnotation

<project-name> .ccr

contains the CCL code.

contains the diagram that corresponds to the .ccl file.

contains the project configuration.

For example, for a project directory named "trades," smart data streaming creates a trades.ccl, trades.cclnotation, and trades.ccr file in the trades directory.

4. Click Finish to create the project files.

The new project opens in the visual editor with one input stream, NEWSTREAM, and an inline schema ready for editing.

Task overview:

About the Portfolio Valuation Sample Project [page 320]

Next task: Editing a Project Diagram [page 325]

Related Information

Creating a Streaming Project in SAP HANA Studio [page 93]

324

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

19.3 Editing a Project Diagram

Edit projects in the visual editor by adding shapes from the palette to the project diagram, connecting them, and completing the configuration of each shape.

Procedure

1. If the sample project diagram is not already open in the visual editor, open it now: a. In SAP HANA Streaming Development perspective, from project explorer, open the sample project, my_portfolio_valuation.

b. Navigate to the .cclnotation file in your project folder and double-click my_portfolio_valuation.cclnotation.

2. Click in the diagram to begin editing using the palette.

Tip

To make the visual editor window full-screen, double-click the

<name>

:Diagram tab at the top. Doubleclick again to revert.

3. Select the input stream element NEWSTREAM that was added automatically when you created the project, right-click, and choose Delete Element.

To run the sample project with example data, delete this element from the project before compiling.

○ Delete Element – removes the element from the project.

○ Delete from Diagram – removes the element from the diagram, but retains it in the project. When you run the project, everything in the project runs, even elements that are not on the diagram.

4. (Optional) To toggle between the visual editor and the CCL editor, choose Switch to Text or Switch to

Visual (F6).

Note

○ Clicking a button or other icon in a shape, or on the main toolbar

○ Using a shortcut key

○ Double-clicking an element to open it

○ Right-clicking to select from the context menu

○ Selecting from the main menu bar

○ Editing element values in the properties view

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Next task: Adding an Input Adapter [page 326]

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

325

19.4 Adding an Input Adapter

Attach an adapter by inserting it in the diagram, connecting it to a stream or window, and setting properties.

Context

This tutorial shows you how to insert an adapter, enable it for schema discovery, then generate and attach the input window and its schema automatically. This is the best practice for creating a schema when using an adapter that supports schema discovery.

Alternatively, SAP HANA smart data streaming allows you to create the stream or window and then attach an adapter. Use this method for adapters that do not support schema discovery, or to explicitly create an inline schema for input streams or windows.

Procedure

1. For this example, from within the SAP HANA Streaming Development perspective, open the Input

Adapters compartment in the palette (to the right of the diagram) and select the File/Hadoop Event

XML Input adapter, which reads data from an XML file.

2. Click the adapter in the palette, then click in the diagram.

The adapter shape is inserted but its border is red, indicating it is not complete, until you define its properties and attach it to a stream or window.

3. In the adapter shape toolbar hovering outside the shape, click Edit Properties ( ) .

4. (Optional) In the Adapter Properties dialog, change Name to identify your adapter.

5. Configure the adapter for schema discovery:

Required properties are in red.

Note

Leave Use named property set unchecked, as this option does not allow you to discover the schema for this adapter.

a. Click in the Value column for Directory and click the Browse button ( ).

b. Click the Browse button in the Directory dialog to select the folder with the data files you want the adapter to read. Click OK.

For this example, specify the absolute path to the sample data installed with the product.

326

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

Property

19.5 Discovering a Schema

In studio, use the Schema Discovery button in the adapter shape to discover and automatically create a schema based on the format of the data from the adapter.

Prerequisites

Add the adapter to the diagram and set its properties.

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

327

Context

Rather than manually creating a new schema in your smart data streaming project, use schema discovery to discover and automatically create a schema, stream, or window based on the format of the data from the datasource to which your adapter connects. In this tutorial, you discover the schema for the PriceFeed input window from the File/Hadoop Event XML Input adapter, which corresponds to the schema of the source file.

Each adapter that supports schema discovery has unique properties that enable schema discovery. For a list

of adapters that support schema discovery and which properties to configure, see Adapter Support for

Schema Discovery [page 104]

Procedure

1. From within the SAP HANA Streaming Development perspective, click Schema Discovery on the adapter toolbar.

Studio displays a Progress Information box and looks for the configuration.

○ If the schema is configured properly and one or more data sets are found, a Schema Discovery: Select

Schema dialog appears where you can view and select a schema.

○ If the schema is not successfully discovered, an error message appears stating that no schema was discovered for the adapter. You can:

○ Check that the adapter properties are configured for schema discovery.

○ Check

Adapter Support for Schema Discovery [page 104]

to see if the adapter supports schema discovery.

2. Select the schema you need.

You can expand the data set to view the schema.

For this example, select positions.xml, then click Next.

3. In the Schema Discovery: Create Element dialog, choose Create new input window (with inline schema).

This option creates and attaches a new window to the adapter, creates an inline schema for the window, and populates the window with the schema discovered from the adapter. When the adapter is not yet attached to a stream or window, other options are the following:

Option

Create a new input stream

(with inline schema)

Create a new input stream

(with attached schema)

Create a new input window

(with attached schema)

Create new named schema

Description

Creates and attaches a new stream to the adapter, creates an inline schema for the stream, and populates the stream with the schema discovered from the adapter.

Creates and attaches a new stream to the adapter, creates and attaches a new named schema to the stream, and populates the stream with the schema discov ered from the adapter.

Creates and attaches a new window to the adapter, creates and attaches a new named schema to the window, and populates the window with the schema discov ered from the adapter.

Creates a new named schema and populates it with the schema discovered from the adapter.

4. Click Finish.

○ The new input window appears with the default name positions_xml_window1, and is automatically connected to the File/Hadoop Event XML Input adapter.

328

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

○ The adapter file property is set. The red warning border disappears, indicating that the element is now valid.

5. In the schema compartment of the input window, click the Toggle Key buttons for the BookId and

Symbol columns to specify the primary key.

The key button indicates primary key columns. With the primary key, the shape becomes valid.

6. Set a keep policy in the input window: a. Click Set Keep Policy .

b. In the Edit Keep Policy dialog, choose All Rows, and click OK.

7. Click the input window Edit button and name it

positions

, using lower case letters.

Next Steps

Create another input window, PriceFeed. Either:

● Create the PriceFeed input window manually, following steps in the next task, or,

● Insert another File/Hadoop Event XML Input adapter and configure it for schema discovery.

This time, when you discover the schema, choose pricefeed.xml in the exampledata directory. Name the input window PriceFeed, click the Id column to make it the primary key, and set the keep policy to

MIN

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Next task: Adding an Input Window Manually [page 329]

19.6 Adding an Input Window Manually

In studio, add an input window to the diagram in the sample project

my_portfolio_valuation

Context

These steps let you create an input window directly, and define the schema, without importing a schema.

If you used the input adapter to discover the schema and generated both input windows automatically, skip these steps and go directly to the next task.

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

329

Procedure

1. From within the SAP HANA Streaming Development perspective, in the visual editor, in the palette to the right of the diagram, open the Streams and Windows compartment.

2. Click Input Window.

3. Click in an empty area in the diagram where you want to insert the input window.

The input window object is added to the project. The red border indicates that it needs more definition to be valid.

4. To set the name of the input window, either:

○ In iconic mode, click once to select the shape, then click again to edit the name.

○ In verbose mode, click the edit icon next to the name.

For this example, enter the name

PriceFeed

5. Click the "plus" sign to expand the shape to verbose mode, if necessary, and click Add Column ( ) on the toolbar in the input window, to add each new column.

Tip

Hover over any icon to see its name.

A new column is created with a default name, and default datatype of integer.

6. Specify additional columns.

a. Double-click each column name to edit it.

b. Double-click each datatype to select the correct datatype.

For this example, enter these column names and datatypes:

Symbol

TradeTime

Price

Shares integer string seconddate float integer

7. Click the button for the Id column to toggle it to the Key symbol.

Input windows require a primary key.

The Id column is now the primary key for the PriceFeed input window. The red warning border disappears, indicating that the element is now valid.

8. Create a retention window.

a. Click Set Keep Policy .

b. In the Edit Keep Policy dialog, select Time, and enter

10 MIN

in the text box. Click OK.

The default policy is to keep all rows of incoming data.

This step defines a KEEP clause, and retains all price events received in the last 10 minutes. Without a

KEEP clause, the PriceFeed window would grow infinitely large. See

Retention Policies [page 29] .

9. Save (Ctrl+S).

This saves changes to both the .cclnotation file (the diagram) and the .ccl file (the CCL).

330

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

Results

The input window and its schema (or deduced schema) are in the diagram.

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Discovering a Schema [page 327]

Next task: Creating an Aggregate as a Simple Query [page 331]

19.7 Creating an Aggregate as a Simple Query

In studio, add an aggregate simple query to the sample diagram to create a volume weighted average price

(VWAP).

Context

An aggregate query groups events that have common values, and computes summary statistics for the group.

Procedure

1. From within the SAP HANA Streaming Development perspective, in the visual editor palette, in Streams and Windows, click Aggregate.

2. Click in the diagram to create the object.

3. Change the default name, Aggregate1, to VWAP.

4. Connect PriceFeed to the VWAP aggregate: a. Click the Connector tool in the palette.

b. Click the PriceFeed input window, then click the VWAP aggregate.

Click the shape that produces the output first, then the shape that receives the data, to indicate the direction of data flow. Watch for visual indicators that show you when the connection is valid:

Indicator Meaning

Connection is allowed

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

331

Indicator Meaning

Connection is not allowed

5. Enter column expressions: a. Click Add Column Expressions ( ), then Copy Columns from Input ( ) in the shape toolbar to select the columns to copy into the schema for the aggregate window.

For this example, copy these columns:

○ PriceFeed.Symbol

○ PriceFeed.TradeTime

○ PriceFeed.Price

b. Edit column names to clarify that these columns will hold the most recent price and time for the group:

○ Change TradeTime to LastTime

○ Change Price to LastPrice c. Add additional columns by clicking Add Column Expression in the shape toolbar.

For this example, add another column and edit its name to

VWAP

6. Edit a column expression by double-clicking to open the inline editor, by selecting one of the tabs in the

Properties view, or by selecting an expression and pressing Ctrl+F2 to edit it using the pop-up expression editor.

For this example, edit the VWAP column expression to: sum ( PriceFeed.Price *

PriceFeed.Shares ) / sum ( PriceFeed.Shares )

7. Click Add Group Clause ({ }) to edit the grouping of columns in the aggregate object.

Note

The aggregate shape must have exactly one GROUP BY expression.

For this example, select PriceFeed.Symbol as the grouping column.

The red warning border disappears, indicating that the element is now valid.

8. Set a keep policy in the input window: a. Click Set Keep Policy .

b. In the Edit Keep Policy dialog, choose All Rows, and click OK.

Results

By default, the aggregate is created as output, which allows external applications to subscribe to or query it, and allows you to view it using the Stream View in the SAP HANA Streaming Run-Test perspective.

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

332

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

Previous task:

Adding an Input Window Manually [page 329]

Next task: Creating a Join as a Simple Query [page 333]

19.8 Creating a Join as a Simple Query

Add a join to the sample project. A join combines events from two or more inputs to create a single stream or window. It is similar to a join in SQL.

Context

Smart data streaming supports inner joins, left and right outer joins, and full outer joins, with join syntax

comparable to SQL ANSI join syntax and comma-separated syntax. For more information, see Joins [page

150]

Procedure

1. From within the SAP HANA Streaming Development perspective, in the visual editor palette, in Streams and Windows, select Join.

If necessary, close the compartments below Streams and Windows, or use the arrow below the compartment, so that Join is visible.

2. Click in the diagram to create the object.

For this example, name the the join object

IndividualPositions

3. Using the connector tool, connect the join object to the appropriate stream or window.

Attach join objects to any stream, window, or Flex operator. Join objects have multiple inputs, but only one output.

Note

Streams, windows, or keyed streams can participate in a join. Only one stream can participate in a join at a time.

For this example, connect the VWAP aggregate object and the Positions input window to the

IndividualPositions join object, in that order.

Tip

To add multiple connections, Shift+click while holding the Connector tool. To return to normal selection, press Esc or click the Select tool in the palette to release it.

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

333

4. Click Add Column Expressions ( ), then Copy Columns ( ) in the join shape toolbar and select columns to copy.

Tip

If you get an error, or do not see all columns from both inputs listed, reconnect the new join element to the Positions or VWAP shapes as needed.

For this example, choose Select All, then clear the check box on VWAP.Symbol so that you do not get the symbol field twice.

5. Click Add Column Expressions ( ).

For this example add two columns: CurrentPosition and AveragePosition.

6. To modify column expressions, do one of the following:

○ Double-click on the expression to open the inline editor, and either type directly or press Ctrl+Space for syntax completion assistance, to pick from column names and functions, or,

○ Press Ctrl+F2 to open the expression editor. Press Ctrl+Space to display the available input columns and built-in functions, or enter the desired expression manually, or,

○ Modify the expression in the Properties view.

For this example, create these Column Expressions:

○ CurrentPosition:

( VWAP.LastPrice * positions.SharesHeld )

○ AveragePosition:

( VWAP.VWAP * positions.SharesHeld )

7. In the Join Conditions compartment of the join shape, set up the join conditions.

If you connected the join to the VWAP and Positions inputs, in that order, there are now two elements in the Join Conditions compartment. The first defines the leftmost element for the join. If you connected to

VWAP first, the first element (left side of the join) is VWAP. For this example, configure the second join element.

a. Double-click the second join element to open the Edit Join Expression dialog.

b. Choose a join type.

For this example, use RIGHT, which is a right outer join. You want RIGHT because VWAP is the first, or left input, and Positions is the second, or right input. You only want your positions in the output; you do not need prices for symbols that are not held in the portfolio.

c. Select the columns to join on.

You cannot edit join constraints manually in the visual editor.

For this example:.

○ As Source 1, ensure that VWAP is in the dropdown, and select the option starting with Symbol: as the column.

○ As Source 2, ensure that Positions is in the dropdown, and select the option starting with Symbol: as the column.

334

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

d. Click Add. If a message displays indicating that datatypes for the selected columns should match, click Yes to continue.

The columns chosen appear in Join Constraints, where you now see:

ON VWAP.Symbol=positions.Symbol

The dialog shows: e. Click OK.

8. Set a keep policy in the input window: a. Click Set Keep Policy .

b. In the Edit Keep Policy dialog, choose All Rows, and click OK.

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

335

9. In the join shape, click (Toggle Type to OUTPUT). If this option is not available, the shape is already set to OUTPUT.

Results

The IndividualPositions join shape now shows the completed join, as shown in the figure.

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Creating an Aggregate as a Simple Query [page 331]

Next task: Preparing Data for Output [page 337]

336

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

19.9 Preparing Data for Output

Add a new aggregate to finalize the data and prepare it for output into SAP HANA.

Context

Create an additional Aggregate Simple Query and name it ValueByBook.

Procedure

1. Connect the IndividualPositions join object to it.

2. Click Copy Columns ( ) in the shape toolbar and copy columns BookId, CurrentPosition, and

AveragePosition.

3. Set column expressions:

○

BookId IndividualPositions.BookId

○

CurrentPosition sum ( IndividualPositions.CurrentPosition )

○

AveragePosition sum ( IndividualPositions.AveragePosition )

Tip

Use the inline editor. Double-click on the column expression, and use the Home and End keys to quickly edit the expression.

4. Add the GROUP BY clause ({ })

IndividualPositions.BookId

5. Toggle to OUTPUT if the shape is not set as OUTPUT already.

Next Steps

Set up a connection and add an output adapter to deposit data into an SAP HANA studio table.

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Creating a Join as a Simple Query [page 333]

Next task: Adding an Output Adapter for SAP HANA [page 338]

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

337

19.10 Adding an Output Adapter for SAP HANA

Create an SAP HANA service definition, and insert and configure an SAP HANA Output adapter using the studio.

Prerequisites

● You have installed the SAP HANA client driver on the same machine as studio and set up an SAP HANA database.

● You have created a connection to the smart data streaming server through the SAP HANA Streaming Run-

Test perspective server view.

Note

The smart data streaming server that you connect to must be version 5.1 SPS 10 or newer. If you connect to an older version, the option to create a HANA service does not appear in the Data Services view.

Procedure

1. In the data services view, right-click on the streaming server and select Discover.

2. Right-click on the workspace you want to use and select Add HANA Service.

3. Right-click the new database service and select Rename Data Service. For this tutorial, name it hanaservice.

4. In the properties view, set the User parameter to the username necessary for communicating with the database.

5. Set the Password parameter to the password for your username.

6. (Optional) Add a description of the database service in the Description parameter.

7. (Optional) Select Use Default HANA Server if you are connecting to the local SAP HANA server.

If you choose to connect to the local SAP HANA server, you need not provide an SAP HANA hostname or instance number.

8. (Required if connecting to a remote SAP HANA database) Specify a database type:

○ Select Single Tenant if the SAP HANA system is the only database at the location, or;

○ Select Multi Tenant if you are connecting to a system with multiple databases. If you select this option, specify the name of the database that you are connecting to in the Database Name field.

9. (Required if connecting to a remote SAP HANA database) Set the HANA Hostname parameter to the fully qualified domain name (FQDN) of the host on which the SAP HANA database is installed.

10. (Required if connecting to a remote SAP HANA database) Set the HANA Instance Number parameter to the instance number of the target database.

338

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

11. (Optional) Enable Multi-Byte Character Support if the database you are connecting to uses multi-byte characters.

12. In the data services view, right-click on the streaming server and select Discover.

13. Right-click on the workspace you want to use and select Add ODBC Service.

14. Click anywhere outside the Properties view to save your work.

15. Open the Output Adapters compartment in the palette and choose HANA Output. This adapter uses multiple parallel ODBC connections to load information from smart data streaming into the SAP HANA server.

a. Add the adapter to the diagram and name it toSAPHANA.

b. Click Edit Properties ( ). Fill in those property values in red:

○ Set Database Service Name to hanaservice.

○ Set Target Database Table Name to the required SAP HANA table. In this example, you will not be configuring the project from the SAP HANA side, so enter exampletable in the value field.

c. Click OK, then press Ctrl+S to save.

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Preparing Data for Output [page 337]

Next task: Preparing to Compile [page 339]

Related Information

Adding an Adapter to a Project [page 96]

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Adapters [page 40]

19.11 Preparing to Compile

Clean up the diagram by removing unused elements.

Procedure

1. Delete any unused elements from the project so that you can run it.

For example, if you have not done so, remove the unused input stream element NEWSTREAM that was added automatically when you created the project.

2. (Optional) Toggle to Iconic mode or Verbose mode:

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

339

○ Click the Toggle Image button in the upper left corner of a shape, or,

○ Click the All Iconic

or All Verbose button in the toolbar.

3. (Optional) Click Layout Left to Right to line up shapes.

4. (Optional) To close the diagram, press Ctrl+W or Ctrl+F4, or click the X on the tab at the top of the editor .

Results

The completed diagram should look like this figure in Verbose mode. You may need to open some compartments and click again to see details for all elements.

Note

Even though it does not appear below, if you added the SAP HANA Output adapter to the project, the adapter is also visible in the diagram.

Figure 8: Completed Sample Portfolio Valuation Diagram

Next Steps

Compile the project and check for errors.

Task overview:

Previous task:

Next task: Compiling the Project and Viewing Problems [page 341]

340

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

19.12 Compiling the Project and Viewing Problems

Compile a project in studio before running it to check for errors and make corrections. Use the Problems view to view error details.

Procedure

1. Update the Default Server URL preference: a. Select Window Preferences .

b. In the Preferences dialog, select SAP HANA smart data streaming.

c. In the Default Server URL field, click Change and select a server from the Select Default Server URL dialog. Click OK.

d. In the Preferences dialog, click Apply, then OK.

2. To compile the project, either:

○ Click the Compile Project

button in the main toolbar, or,

○ Press F7.

The project compiles and reports any errors found. Compilation errors are displayed in the Problems or

Console view, depending on the type of error.

3. Click on a problem in the Problems view, or expand the group to see individual errors.

By default, the Problems view is at the bottom of the screen, and problems are grouped by severity.

Error details appear in the Problems view and in the status bar at the bottom left side of the screen.

Tip

If you double-click on a problem in the Problems view while the project is open in the visual editor, the

CCL editor opens read-only to show you where the problem is. To fix the problem, either:

○ Return to the visual editor and fix it there, or,

○ Close both the visual editor and CCL editor for the project, and then reopen the project in the CCL editor.

4. If the error message is too long to show the entire message, click it to read the full text in the status bar at the bottom of the studio window.

5. Right-click an item to choose from the context menu:

Option Action

Go to

Copy

Highlight the problem in the .ccl file. The CCL editor opens in read-only mode.

Copy error details to the clipboard. When you exit studio, the contents of problems view are removed.

Use this option to save off errors.

Show in Display details in Properties view.

Quick Fix (Disabled)

Properties Display details in a dialog box.

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

341

6. (Optional) Click the View dropdown menu to see more options.

7. Click the Console tab to view compiler results.

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Preparing to Compile [page 339]

Next task: Deploying the Sample Project [page 342]

19.13 Deploying the Sample Project

Run the project and watch it open in the SAP HANA Streaming Run-Test perspective.

Prerequisites

Make sure the project compiles without errors. Correct any problems before you run the project.

Procedure

1. With the diagram open in the visual editor in the SAP HANA Streaming Development perspective, click in the main toolbar.

2. Review the running project in SAP HANA Streaming Run-Test perspective.

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Compiling the Project and Viewing Problems [page 341]

Next task: Loading Data into the Sample Project [page 343]

342

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

19.14 Loading Data into the Sample Project

Test the sample project by loading reference data into the Positions window.

Context

If your project has the File/Hadoop Event XML Input adapter attached to the Positions input window, data is loaded automatically when you start the project. If you removed or omitted the adapter, use this alternative process to load the sample data.

Procedure

1. From within the SAP HANA Streaming Run-Test perspective, in the server view, expand the my_portfolio_valuation project to show the list of windows and streams in the project.

2. Double-click the Positions input window to open it in stream view.

Stream view is in the upper right-hand portion of the SAP HANA Streaming Run-Test perspective.

○ If your diagram has the File/Hadoop Event XML Input adapter connected to the Positions input window, stream view shows sample data for Positions, loaded automatically from the adapter.

○ If you removed the adapter, go to the next step to load the data manually.

3. Load positions data from a file into the Positions window.

a. Go to file upload view.

b. Click the Select Project button in the view toolbar. If you only have one project open in server view, no dialog appears and smart data streaming automatically selects the project for you. Otherwise, select the my_portfolio_valuation project in the dialog, and click OK.

c. Click the Browse button, navigate to your

<SAP HANA-studio-workspace> \exampledata folder, and select positions.xml.

When you install the SAP HANA smart data streaming package, you install the

SAP_HANA_STREAMING/examples/exampledata folder with it. If you have not done so already, copy the exampledata folder to your SAP HANA studio workspace.

d. Click Open.

e. With positions.xml highlighted in File Upload view, click the Upload button.

Watch the data flow in stream view, as studio loads the three positions for Book1 and Book2.

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Deploying the Sample Project [page 342]

Next task: Testing the Project with Recorded Data [page 344]

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

343

19.15 Testing the Project with Recorded Data

Play back the previously recorded price feed data, and view the continuous portfolio valuations in the sample project.

Procedure

1. From within the SAP HANA Streaming Run-Test perspective, in the server view, double-click the

IndividualPositions, VWAP, and ValueByBook output windows.

In the server view list, a red arrow in the lower right corner of the window icon ( ) indicates the output windows.

2. Click the Playback tab.

3. If necessary, click the Select Project button in the upper right corner of playback view.

○ If you only have one project running, studio selects it for you.

○ Otherwise, select the my_portfolio_valuation project in the dialog and click OK.

4. Click the Select Playback File button.

5. Navigate to your

<SAP HANA-Studio-workspace> \exampledata folder, and select pricefeed.xml.

Click Open.

When you install the SAP HANA smart data streaming package, you install the SAP_HANA_STREAMING/ examples/exampledata folder with it. If you have not done so already, copy the exampledata folder to your SAP HANA studio workspace.

6. In the playback view, in the playback mode frame, click the rec/ms button, then enter a rec/ms value of 1.

A value of 1 plays back at a rate of 1000 records per second.

7. Click the green Start Playback button to start playback of the price feed.

8. While the data plays back, click each of the output windows in stream view to see the calculations revised in real-time.

9. (Optional) In the event tracer view, choose Select Running Project, and click Initialize with Base Data.

In this example, Event Tracer shows the PriceFeed and Positions elements in green to indicate Insert operations. VWAP, IndividualPositions, and ValueByBook are in blue, indicating Updates. Colors change as different event types are processed.

Double-click each node to watch event data in the Console.

344

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

10. To stop the playback, click Stop .

11. When you are done testing the project, right-click it in server view and choose Stop Project.

If you omit this step, the project stops when you exit studio, but you may get an error.

Tip

If you see an error when you restart studio, or when you try to open a .ccl file after running a project, there may be multiple instances of studio trying to use the same studio workspace location. If this occurs, close studio and restart it.

Task overview:

Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Loading Data into the Sample Project [page 343]

Next: Sample Project with Modules [page 346]

Related Information

Project Execution and Testing [page 79]

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

345

19.16 Sample Project with Modules

SAP HANA Output adapter is not included in this module.

The module, valuation.ccl, computes the VWAP aggregate, and creates a join to the Positions window. The project uses the module to divide the moving data into smaller partitions, based on the first letter of the

Symbol column. This strategy spreads the load out to more cores, thereby increasing throughput. By using modules, with very little coding you can easily double, quadruple, and so on, the number of partitions.

This example also implements the streaming tick data in PriceFeed as a stream rather than as an input window. Because keeping every tick would use a lot of memory, and because the state is never updated or queried, a stream is a more likely choice than a window in a real-world scenario for this event stream.

Example

Create Module valuation

The valuation module:

1. Defines the input stream TradesIn.

2. Defines a stream, Filter1, that filters TradesIn data into a substream based on the declared parameters afrom and ato.

3. Defines the input window Portfolio.

4. Defines the VWAP aggregate as an output window.

5. Defines another output window, ValueBySymbol, that performs a join similar to the join simple query in the simple PortfolioValuation project, with the addition of a cast for the float data.

CREATE MODULE valuation

IN TradesIn,Portfolio

OUT ValueBySymbol, VWAP

BEGIN

IMPORT 'import.ccl';

DECLARE

PARAMETER STRING afrom;

PARAMETER STRING ato;

END;

CREATE INPUT STREAM TradesIn

SCHEMA TradesSchema ;

CREATE STREAM Filter1 AS

SELECT * FROM TradesIn

WHERE substr(TradesIn.Symbol,1,1) >= afrom

and substr(TradesIn.Symbol,1,1) <= ato

;

CREATE INPUT WINDOW Portfolio

SCHEMA PortfolioSchema

PRIMARY KEY (BookId, Symbol);

CREATE OUTPUT WINDOW VWAP

PRIMARY KEY DEDUCED AS

SELECT Filter1.Symbol Symbol ,

(sum((Filter1.Price * cast(FLOAT ,Filter1.Shares))) /

cast(FLOAT ,sum(Filter1.Shares)))

AS VWAP,

sum (Filter1.Shares ) Total_Shares ,

valueinserted(Filter1.Price) LastPrice,

valueinserted(Filter1.TradeTime) TradeTime

346

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

FROM Filter1

GROUP BY Filter1.Symbol ;

CREATE OUTPUT WINDOW ValueBySymbol

SCHEMA (BookId STRING, Symbol STRING, CurrentPosition FLOAT,

AveragePosition FLOAT)

PRIMARY KEY (BookId, Symbol) AS

SELECT

Portfolio.BookId AS BookId,

Portfolio.Symbol AS Symbol,

(VWAP.LastPrice * cast(FLOAT ,Portfolio.SharesHeld))

AS CurrentPosition,

(VWAP.VWAP * cast(FLOAT ,Portfolio.SharesHeld))

AS AveragePosition

FROM Portfolio JOIN

VWAP

ON Portfolio.Symbol = VWAP.Symbol;

END;

Example

Create Named Schema TradesSchema

CREATE SCHEMA TradesSchema

( Id integer ,

Symbol string ,

TradeTime seconddate ,

Price float ,

Shares integer ) ;

Example

Create Named Schema PortfolioSchema

CREATE SCHEMA PortfolioSchema

( BookId string ,

Symbol string ,

SharesHeld integer ) ;

Example

Import and Load the valuation Module

In the parent scope, the valuation module is loaded three times, as Valuation1, Valuation2, and Valuation3.

1. The IN clause binds the input streams in the module to streams in the parent scope. TradesIn is bound to InputStream1, and Portfolio is bound to InputPositions.

2. The OUT clause binds the output window in the module, ValueBySymbol, with the three parameterized output windows, VbySym1, VbySym2, and VbySym3, and partitions the VWAP aggregate as VWAP1,

VWAP2, and VWAP3.

InputStream1

InputPositions

UnionVWAP

Input stream based on the imported schema, TradesSchema.

Input window based on the imported schema, PortfolioSchema.

Output window created as a UNION of the partitioned VWAP aggregate.

IMPORT 'import.ccl';

IMPORT 'valuation.ccl';

DECLARE

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

347

PARAMETER STRING afrom :='A';

PARAMETER STRING ato := 'Z';

END;

CREATE INPUT STREAM InputStream1 SCHEMA TradesSchema ;

CREATE INPUT WINDOW InputPositions

SCHEMA PortfolioSchema PRIMARY KEY ( BookId , Symbol ) ;

LOAD MODULE valuation as Valuation1

in TradesIn = InputStream1, Portfolio = InputPositions

OUT ValueBySymbol = VbySym1, VWAP = VWAP1

PARAMETERS afrom = 'A', ato = 'J'

;

LOAD MODULE valuation as Valuation2

in TradesIn = InputStream1, Portfolio = InputPositions

OUT ValueBySymbol = VbySym2, VWAP = VWAP2

PARAMETERS afrom = 'K', ato = 'Q'

;

LOAD MODULE valuation as Valuation3

in TradesIn = InputStream1, Portfolio = InputPositions

OUT ValueBySymbol = VbySym3, VWAP = VWAP3

PARAMETERS afrom = 'R', ato = 'Z'

;

CREATE OUTPUT WINDOW UnionVWAP

PRIMARY KEY DEDUCED

AS SELECT * FROM VWAP1

UNION SELECT * FROM VWAP3

UNION SELECT * FROM VWAP2 ;

CREATE OUTPUT WINDOW ValueBySymbol

PRIMARY KEY (BookId,Symbol)

AS SELECT * FROM VbySym1

UNION SELECT * FROM VbySym3

UNION SELECT * FROM VbySym2 ;

// ----------------------------

// stream ValueByBook

CREATE OUTPUT WINDOW ValueByBook

SCHEMA (BookId STRING, CurrentPosition FLOAT, AveragePosition FLOAT)

PRIMARY KEY DEDUCED AS

SELECT ValueBySymbol.BookId AS BookId,

sum(ValueBySymbol.CurrentPosition) AS CurrentPosition,

sum(ValueBySymbol.AveragePosition) AS AveragePosition

FROM ValueBySymbol

GROUP BY ValueBySymbol.BookId;

ATTACH INPUT ADAPTER Adapter1 TYPE toolkit_file_xmllist_input TO InputStream1

GROUP nostartGroup

PROPERTIES dir = '../exampledata' , file = 'pricefeed.xml' , pollingPeriod = 0 , xmllistSecondDateFormat = 'yyyy-MM-ddTHH:mm:ss' , xmllistMsDateFormat = 'yyyy-MM-ddTHH:mm:ss';

ATTACH INPUT ADAPTER Adapter2 TYPE toolkit_file_xmllist_input TO InputPositions

PROPERTIES dir = '../exampledata' , file = 'positions.xml' , pollingPeriod = 0 , xmllistSecondDateFormat = 'yyyy-MM-ddTHH:mm:ss' , xmllistMsDateFormat = 'yyyy-MM-ddTHH:mm:ss';

ADAPTER START GROUPS nostartGroup nostart ;

Note

Edit the default value for the dir property to the sandbox base directory. For example, change dir='../exampledata to dir=' <sandbox-base-directory> /exampledata'. Otherwise, the example does not compile properly and you receive an error message.

348

P U B L I C

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

Parent topic: Appendix: Tutorial for Building and Testing a Project in Studio [page 317]

Previous task:

Next: Additional Sample Projects [page 349]

Related Information

About the Portfolio Valuation Sample Project [page 320]

Modularity [page 185]

19.17 Additional Sample Projects

Smart data streaming in SAP HANA studio includes several completed example projects.

Customize the Welcome page in SAP HANA studio to show samples of completed streaming projects. Click the Customize Page icon in the top right corner of the Welcome page to open the Customize dialog. Select the

Home tab and in the Root Pages section, select the Samples option and click OK. The Welcome page now displays a Samples subpage.

You can view the examples in studio and run them against sample data installed with the product. Stepping through examples is an ideal way to watch a simplified set of event data flow through the system.

The examples include:

Indexes

Calculation

Shows how continuous computations can be applied to a stream of market prices to deliver insight into the market. This example demonstrates reusable modules. Each of the market calculations is defined in an external module, that is, a module defined in a separate CCL file, and then imported into the project. Parameters (in this case, time and intervals) are set when the module is called.

Pattern

Matching

Provides a simple example of situation detection: watching for a pattern of events. The example demonstrates how to watch for employee fraud in a retail setting, based on transaction patterns from a point-of-sale system. The example applies three filter queries to an input stream of transactions, and then uses a pattern query (CCL MATCHING clause) to produce a possible fraud alert event when all of the criteria occur in the defined time interval.

Prepay Biller Loads call events (CDRs), such as those generated from a telephone carrier network, and applies account and call plan information to create a billing record for each call, and maintain a balance of prepaid minutes. The example demonstrates joins, aggregation, and using joins to augment the events with reference data.

Top 3 Prices Creates a window showing the top three distinct trade prices for each symbol. The example uses a Flex operator to create a custom operator with an embedded CCLScript script. A Flex operator creates a single output stream or window, and allows greater flexibility and control

SAP HANA Smart Data Streaming: Developer Guide

Appendix: Tutorial for Building and Testing a Project in Studio

P U B L I C

349

VWAP than a simple SELECT statement. The example also uses a named schema, which can be defined once and shared by the input stream and output window.

Defines an input stream of stock market prices, as they might be reported from an exchange, and computes a moving average price called the volume weighted average price (VWAP).

Uses a filter, and a simple aggregation (GROUP BY).

For details of each example, see the Samples subpage on the Welcome page.

See the SAP HANA Smart Data Streaming: CCL Reference for more code samples that demonstrate CCL and

CCLScript.

In this section:

Loading a Sample Project [page 350]

Load and run one of the example projects installed with smart data streaming to view end-to-end project execution in your workspace.

Parent topic: Appendix: Tutorial for Building and Testing a Project in Studio [page 317]