KPS and API Gateway - Oracle Help Center

KPS and API Gateway - Oracle Help Center
 Oracle® Fusion Middleware
Oracle API Gateway Key Property Store User Guide
11g Release 2 (11.1.2.4.0)
July 2015
Oracle API Gateway Key Property Store User Guide, 11g Release 2 (11.1.2.4.0)
Copyright 1999, 2015, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.22719, Commercial Computer Software License (December 2007). Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065. This software is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure the safe use of this software. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software in dangerous applications. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. This software and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services. This documentation is in prerelease status and is intended for demonstration and preliminary use only. It may not be specific to the hardware on which you are using the software. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to this documentation and will not be responsible for any loss, costs, or damages incurred due to the use of this documentation. The information contained in this document is for informational sharing purposes only and should be considered in your capacity as a customer advisory board member or pursuant to your beta trial agreement only. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described in this document remains at the sole discretion of Oracle. This document in any form, software or printed matter, contains proprietary information that is the exclusive property of Oracle. Your access to and use of this confidential material is subject to the terms and conditions of your Oracle Software License and Service Agreement, which has been executed and with which you agree to comply. This document and information contained herein may not be disclosed, copied, reproduced, or distributed to anyone outside Oracle without prior written consent of Oracle. This document is not part of your license agreement nor can it be incorporated into any contractual agreement with Oracle or its subsidiaries or affiliates. 27 July 2015
Contents
Preface
6
Who should read this document
6
How to use this document
6
1 Introduction to KPS
8
KPS architecture
8
KPS data stores
9
KPS client applications
9
When to use a KPS
9
2 Get started with KPS
10
Example KPS table
10
Before you begin
11
Define KPS configuration with Policy Studio
11
Step 1: Define where data will be stored
11
Step 2: Define the KPS table
12
Step 3: Define a policy that accesses the table
14
Step 4: Deploy the configuration
16
Add KPS data using API Gateway Manager
17
Access KPS data from a policy
18
Enable API Gateway tracing
19
3 Configure KPS in Policy Studio
20
Configure a KPS collection
20
Configure a KPS table
21
KPS aliases
22
KPS data sources
22
KPS table structure
22
Query tables using properties and keys
23
Primary key
23
Secondary key
24
Auto-generated properties
24
Encrypted properties
24
4 Access KPS data using selectors
26
KPS selector syntax
26
KPS selector examples
27
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 3
5 Manage a KPS using kpsadmin
Start kpsadmin
28
28
Start in verbose mode
28
Select kpsadmin operations
28
Table operations
29
Table administration operations
29
Collection administration operations
30
Cassandra administration operations
31
General administration operations
31
Example switching data source
31
Step 1: Backup collection data using kpsadmin
31
Step 2: Create a new data source
32
Step 3: Deploy the configuration
33
Step 4: Restore collection data using kpsadmin
33
6 Configure Apache Cassandra KPS storage
34
Cassandra configuration
34
Cassandra ports
35
cassandra.yaml
35
client.yaml
35
jvm.xml
36
Secure ports
36
Cassandra logging
36
Cassandra configuration steps
37
Step 1: Configure the Cassandra topology
37
Step 2: Start API Gateways with embedded Cassandra servers
43
Step 3: Configure the Cassandra replication factor
43
Step 4: Configure request consistency levels
45
Disable Cassandra storage
7 Configure database KPS storage
Shared database storage
45
47
47
Step 1: Create a KPS database table
47
Step 2: Set up an external connection to the database
48
Step 3: Use the external connection in a KPS collection
48
Per-table database storage
51
Map a database table using a single key
51
How to map a database table using a composite key
56
8 Configure file-based KPS storage
Configure a file-based KPS collection
Appendix A: KPS FAQ
KPS and API Gateway
Oracle API Gateway 11.1.2.4.0
59
59
60
60
Key Property Store User Guide 4
What is KPS used for in API Gateway?
60
What is KPS not suitable for?
60
What are the transaction semantics of KPS?
60
What is the KPS collection alias prefix for?
60
Why are some collections hidden? How can you show them?
61
How do I change API Gateway group passphrase?
61
KPS storage options
62
Is Apache Cassandra storage required? Can you use file or database?
62
How do you switch storage for a KPS collection?
62
Why use database storage?
62
Why use file storage?
62
When can you use kpsadmin? When should you use storage-specific tools?
63
Apache Cassandra
63
Why use Cassandra as a KPS storage option?
63
What version of Cassandra does the API Gateway use?
63
What does all host polls marked down mean?
63
Can you use an external Cassandra instance?
63
How do you set Cassandra consistency levels?
64
How do you disable Cassandra?
64
Appendix B: Troubleshoot KPS error messages
All platforms
65
65
All host polls marked down
65
Nodetool reports failed to connect
65
May not be enough replicas present to handle consistency level
66
Windows only
66
Node running in client mode
66
UTF-8 characters not displaying correctly in kpsadmin
67
FSUTIL utility bug
67
Path length issue >255 characters
67
Could not drop kps keyspace using cassandra-cli
68
Appendix C: Apache Cassandra operations for API Gateway
69
nodetool repair
69
Backup and recovery
69
Back up and restore Cassandra node data
69
Back up API Gateway KPS configuration
70
Back up API Gateway Cassandra configuration and data
70
Replace dead node
70
Further information
72
Glossary
73
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 5
Preface
This document describes how to configure and manage the API Gateway Key Property Store (KPS). The KPS enables you to manage API Gateway data referenced from policies running on the API Gateway.
Who should read this document
The intended audience for this document is KPS administrators and policy developers. For more details on API Gateway user roles, see the API Gateway Concepts Guide. This document assumes that you are familiar with the following:
l Database concepts such as tables, rows, and keys
l API Gateway configuration and deployment
l API Gateway selectors
l Using command line tools l Database configuration where database storage is required
For more details on API Gateway configuration and selectors, see the API Gateway Policy Developer Guide.
How to use this document
This document should be used with the other documents in the API Gateway documentation set. Before you begin, review this document thoroughly. The following is a brief description of the contents of each chapter:
l Introduction to KPS on page 8 provides an overview of the KPS architecture and features.
l Get started with KPS on page 10 explains how to develop an example KPS table for managing simple user information.
l Configure KPS in Policy Studio on page 20 provides more detail on how to define general KPS configuration using the P olicy Studio graphical tool.
l Access KPS data using selectors on page 26 explains how to access data in policies on the API Gateway at runtime. l Manage a KPS using kpsadmin on page 28 explains how to manage a KPS, independent of data source.
l Configure Apache Cassandra KPS storage on page 34 explains how to store KPS data in the default Cassandra server embedded in the API Gateway.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 6
Preface
l Configure database KPS storage on page 47 explains how to store KPS data in a relational database (for example, Oracle, MySQL, IBM DB2, or Microsoft SQL Server).
l Configure file-based KPS storage on page 59 explains how to store KPS data in a directory on the file system.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 7
Introduction to KPS
1 A Key Property Store (KPS) is a table of data that can be referenced from policies running on the API Gateway. Data in a KPS table is assumed to be read frequently and seldom written, and can be changed without incurring an API Gateway service outage. KPS tables are shared across an API Gateway g roup. KPS architecture
The following diagram shows a simple role-based architecture:
A KPS is typically used to store property values that are used in p olicies running on an API Gateway. KPS data is injected into policies using selectors that are first created in Policy Studio by policy developers. Selectors are evaluated and expanded dynamically at runtime. For example, a KPS table could contain authorization tokens for different users. A policy could look up the token for the current user and insert it into an HTTP request. KPS tables are organized into collections. The tables in a collection typically have some sort of relationship to one another. For example, the OAuth collection contains a set of tables that store all OAuth-related data. Every KPS table is assigned an alias so that it c an be easily referred to in a policy or a REST request. KPS c ollections and tables can be created by policy developers using P olicy Studio. KPS administrators can use the API Gateway Manager web console to view and modify KPS data at runtime. This is a business or operational role that manages dynamic policy configuration data in a KPS (for example, customer details, authorization levels, or quotas). This means that this information does not need to be configured at design time by policy developers.
For more details on API Gateway architecture, components, and roles, see the API Gateway Concepts Guide. Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 8
1 Introduction to KPS
KPS data stores
KPS data can be stored in one of the following locations:
l Embedded Apache Cassandra database: Data can be distributed across multiple nodes to p rovide high availability (this is the default KPS data store).
l Relational database: Accessible to all API Gateway instances in the API Gateway group.
l JSON files: On the local file system.
KPS client applications
API Gateway p rovides the following client applications: l Policy Studio: Enables policy developers to create KPS collections and tables, and to configure data sources. l API Gateway Manager: Includes a visual web-based interface to enable KPS administrators to view and modify KPS data at runtime.
l kpsadmin c ommand: Supports KPS data entry and other administrative functions. It is designed for use in a development environment.
l KPS REST API: Enables remote p rogrammatic clients to read and write KPS data.
When to use a KPS
KPS provides a flexible data storage service that can be used to store any configuration data. Its primary function is to make this d ata available to selectors at runtime, and it is optimized for this purpose. This makes it most suitable for data with the following c haracteristics:
l Data is common to all API Gateways in an API Gateway group. KPS is not suitable for data that is specific to one particular API Gateway. l The data schema is relatively simple. Each KPS table is assumed to be independent of all others, and referential integrity across tables is not supported.
l Data can change while API Gateways are running. Updating Cassandra-backed or databasebacked KPS tables does not require an API Gateway restart. Changeable data should therefore be stored in KPS instead of hard-coded into p olicies. l Queries always involve looking up a key value in a table to retrieve a single object. This is the usage model supported by selectors. Ad hoc queries that involve searching for non-key properties are not supported.
l Multi-operation transactions are not required. Each read or write to a KPS table is considered a standalone operation. Locking or rollback across multiple operations are not supported.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 9
2 Get started with KPS
This topic explains how to develop an example KPS table for managing simple user information. Example KPS table
The final structure of the example table is displayed in Policy Studio as follows:
This table structure is described as follows:
Column
Type
Description
age
java.lang.Integer
User age.
email
java.lang.String
User email address. This is selected in Policy Studio as a unique Primary Key, which is indexed implicitly by default.
firstName
java.lang.String
User first name.
lastName
java.lang.String
User surname.
password
java.lang.String
User password, which is selected as Encrypted in Policy Studio.
Example table data is displayed on the Settings > Key Property Stores tab in the API Gateway Manager web console:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 10
2 Get started with KPS
Before you begin
The following prerequisite steps apply to this example:
1. Ensure that an API Gateway and an Admin Node Manager are running. 2. Start Policy Studio, and connect to the Admin Node Manager.
For more details, see the API Gateway Installation Guide.
Define KPS configuration with Policy
Studio
The main steps for configuring KPS tables in Policy Studio are as follows:
1. Define where the data will be stored.
2. Define the KPS table.
3. Define a policy that accesses the table.
4. Deploy the configuration.
Step 1: Define where data will be stored
You must first create a KPS collection in which to store the KPS table configuration. Perform the following steps:
1. In the Policy Studio tree, sselect Key Property Stores, and select Add KPS Collection. 2. In the Add KPS Collection dialog, name the collection Samples. Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 11
2 Get started with KPS
3. Select a Default data source of Embedded (Cassandra) for all tables in the collection. Note
Leave the Alias prefix field blank. Step 2: Define the KPS table
To create a table, perform the following steps: 1. In the Policy Studio tree, right-click the newly-created Samples collection, and select Add
Table:
2. In the dialog, enter a Name of User, and provide a Description.
3. Click Add to assign an alias of User to this table. A table must have at least one alias.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 12
2 Get started with KPS
4. Next define the table structure. This consists of the table columns and the data type stored in each column. Select the User table and Structure tab, and click Add: 5. Repeat to add the following columns for your table structure in the Add Property dialog: l email l password l firstName l lastName l age Note
age has an Integer (numeric) Type. All the other columns are String.
5. When you select the User table, you should have the following structure:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 13
2 Get started with KPS
6. You want the email field to be the primary key for the table, so select Primary Key for this field.
7. You want the password field to be encrypted when stored in the data source, so select Encrypted for this field.
Step 3: Define a policy that accesses the
table
To define a test policy that accesses the table, perform the following steps:
1. Add a test policy with a Set Message filter from the Conversion filter category.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 14
2 Get started with KPS
2. Right-click the filter, and set it as the Start filter for the policy. 3. Enter a filter Content-Type of text/plain. 4. Enter the following Message Body for use in the policy:
========================
User
===
Email: ${kps.User[http.querystring.id].email}
First Name: ${kps.User[http.querystring.id].firstName}
Last Name: ${kps.User[http.querystring.id].lastName}
Age: ${kps.User[http.querystring.id].age}
========================
These settings are displayed as follows in the Set Message filter:
The message body value are specified using selectors, which are evaluated and expanded dynamically at runtime. For example, the user age is specified using the following selector string: ${kps.User[http.querystring.id].age}
The selector parts are explained as follows:
Selector part
Description
${
Indicates the start of the selector using a { bracket.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 15
2 Get started with KPS
Selector part
Description
kps
Specifies that selector should query a KPS table.
.User
Specifies the alias of the table to query (in this case, User).
[
Indicates the start of a table property reference using a[ bracket.
http.querystring.id
This is a dynamic query based on an HTTP query string parameter of id. The primary key value is retrieved from this parameter. The row with this key value is returned from the User table if it exists.
]
Indicates the end of a table property reference using a ] bracket.
.age
Retrieves the age column.
}
Indicates the end of the selector using a } bracket.
Note
Add a Reflect Message filter (Conversion category) to return a successful HTTP response status of 200.
5. Set up a path to this policy. In this example, the path is /kpsGetViaSelector:
Step 4: Deploy the configuration
When you are finished with your configuration changes, you must deploy them to the API Gateway. To deploy the new configuration, click Deploy in the Policy Studio toolbar:
This pushes the configuration to the API Gateway group.
Tip
If you deploy an incorrect configuration (for example, specify an incorrect primary key, property type, or name) you can use the kpsadmin command to drop the table in storage. For more details, see Manage a KPS using kpsadmin on page 28.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 16
2 Get started with KPS
Add KPS data using API Gateway Manager
You can use API Gateway Manager to populate the User table with data. Perform the following steps:
1. To access API Gateway Manager in your browser, go to https://localhost:8090.
2. Select the Settings > Key Property Stores tab. 3. Select the Samples > User table.
4. To enter new records, select Actions > New Entry:
5. Click Save to save a record.
For example, the table should look as follows:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 17
2 Get started with KPS
Access KPS data from a policy
For example, to access KPS data from a policy, go to the following URL in your browser:
http://localhost:8080/[email protected]
This URL specifies the user ID (email) as [email protected]
You must specify an email that exists in your data. For example:
Note
If you enter an email that does not exist, you will see [invalid field] results. For example:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 18
2 Get started with KPS
Enable API Gateway tracing
To enable API Gateway debug tracing, perform the following steps in Policy Studio:
1. In the tree on the left, select Server Settings > General.
2. Select a Tracing level of DEBUG. 3. Click Save.
4. Click Deploy. Note
This setting enables debug tracing for the entire API Gateway, and not just for the KPS.
For more details on API Gateway tracing and logging, see the API Gateway Administrator Guide.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 19
3 Configure KPS in Policy
Studio
This topic describes how to define general KPS configuration in P olicy Studio. For details on data source-specific configuration, see the following topics: l Configure Apache Cassandra KPS storage on page 34
l Configure database KPS storage on page 47
l Configure file-based KPS storage on page 59
Configure a KPS collection
A KPS collection is a set of related KPS tables. To configure a KPS collection, perform the following steps:
1. Right-click Key Property Stores in the Policy Studio tree, and select Add Key Property
Store.
2. Specify the following settings in the dialog: l Name: A collection must have a unique name in the API Gateway g roup. l Description: You can provide an optional description. l Alias prefix: You c an also specify an alias prefix, but in normal usage, you c an leave this field blank. l Default data source: A collection has a default data source where all data for all tables in the collection is stored. You can change this data source or assign a different data source to individual tables in the c ollection. 3. When the collection is created, you can specify a Cache for storage and retrieval of selector results. This will improve selector read p erformance for storage backends such as databases. For more details o n API Gateway c aching, see the API Gateway Policy Developer Guide
Note
Only local caches are supported.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 20
3 Configure KPS in Policy Studio
The following shows a KPS collection created in Policy Studio:
Configure a KPS table
A KPS table is a user-defined table managed by the KPS in the API Gateway. To configure a KPS table, perform the following steps:
1. Right-click a KPS collection in the Policy Studio tree, and select Add Table.
2. Specify the following settings in the dialog: l Name: A KPS table must have a unique name in the collection. l Description: You can provide an optional description. l Override the default data source with the following: You can specify a different data source than the collection if required. 3. When the table is created, if required, you can use the Override the default data source
with the following setting to specify a different data source than the collection: Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 21
3 Configure KPS in Policy Studio
4. Finally, click the Structure tab, and click Add to define the structure of the data stored in the table. For details on supported types and keys, see the following:
l KPS table structure on page 22
l Query tables using properties and keys on page 23
KPS aliases
KPS tables are accessed by alias. A table must have at least one alias. Aliases must be unique in an API Gateway g roup. You also can use the optional alias prefix for the collection to help ensure that the alias is unique. The full alias of a table is the collection alias prefix and the table alias combined. For example, samples and User g ives samplesUser . If unspecified, the default value of the alias prefix for the collection is an empty string (for example, User o nly). KPS data sources
A KPS collection has one active data source associated with it. All tables in the collection use this data source by default. You can c onfigure a table to use a different data source if required (see Configure a KPS table on page 21).
KPS table structure
When creating a table, you must define the structure of the data that is stored in that table. This consists of property (column) names and types. You can choose from the following types.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 22
3 Configure KPS in Policy Studio
Type
Description
String
Java type.
Boolean
Byte
Integer
Long
Double
List
Java List of any one of the above Java types. Map
Java Map. The key can be any one of the above Java types. The value can be any one of the above Java types. Query tables using properties and keys
You can directly access records in a table by specifying certain p roperty names and values, without needing to read all the records in the table. You can use only properties of String, Long, and Integer types for this purpose. These properties are known as indexable properties. Indexed properties include primary keys, secondary keys (which are indexed implicitly), and o ther properties that you explicitly select as Indexed in Policy Studio.
Primary key
You can directly access any record using its primary key. All records in the table must be accessible using a unique primary key. You must select one Primary Key per table in Policy Studio. The specified property must be an indexable property. Primary key values cannot be null. Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 23
3 Configure KPS in Policy Studio
Secondary key
You can optionally access any record directly using a unique secondary key. The secondary key can be a simple key (for example, email ) or a composite key (for example, appId o r companyId ) . The specified properties must be indexable properties. The secondary key (and parts of a composite secondary key) cannot be null. You can specify one secondary key per table. A common use case is to specify an internal unique ID as as primary key, and an external user facing ID as a secondary key. For example, id for internal primary key, and email for external ID as a secondary key. Selector access
You can access records in a KPS table using an API Gateway selector. If a secondary key is defined for the table, you must specify all secondary key values in the selector. If no secondary key is defined, you must specify the primary key value instead. In Policy Studio, you can specify a secondary key or a primary key in the Use the following property name(s) for looking up a
table from a selector field.
For examples of accessing KPS tables using selectors, see the following: l Get started with KPS on page 10
l Access KPS data using selectors on page 26
l Configure database KPS storage on page 47
Auto-generated properties
In Policy Studio, you can select that String fields are Autogenerated. When a record is created, a Java java.util.UUID is assigned to the field if it is empty. Note
Values f or auto-generated fields can be supplied and modified by users. KPS o nly generates a value at creation time if no value is already p resent.
Encrypted properties
In Policy Studio, you can select that String fields are Encrypted in storage. However, fields selected as Indexed (including primary and secondary key fields) cannot be encrypted. You can enter values for encrypted fields using the API Gateway Manager or the kpsadmin c ommand. These values are forwarded to the API Gateway in the clear using the KPS REST service, and encrypted before being written to storage. Note
The KPS REST service must always run over HTTPS (the default). You must set an encryption passphrase for the API Gateway group, because this is used in the encryption process. For more details, see the API Gateway Administrator Guide.
When KPS tables are accessed using API Gateway selectors at runtime, encrypted fields are automatically decrypted. Selectors do not need to be aware that particular fields in a table are encrypted in storage. Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 24
3 Configure KPS in Policy Studio
When KPS tables are read using the REST API, data is always returned in its encrypted state. Sometimes you may need to view decrypted data to help debug problems on an API Gateway. You can do this using debug mode in kpsadmin. This requires you to enter the passphrase for the API Gateway group. If the in-built KPS encryption mechanism does not suit your needs, you can encrypt and d ecrypt data outside the KPS. In this case, you should not select properties in KPS tables as encrypted in Policy Studio. Encrypted data must be string-encoded for storage (for example, base64-encoded). Selectors that access the data must decrypt it themselves (for example, using a d edicated d ecryption
filter in Policy Studio). Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 25
4 Access KPS data using
selectors
Data in KPS tables can be accessed using selectors that execute in policies on the API Gateway at runtime. This topic explains KPS selector syntax and provides some example selectors. KPS selector syntax
KPS selector syntax is as follows:
${kps.alias[key].property
The parts in the selector are described as follows:
Selector
part
Description
${
Indicates the start of the selector using a { bracket.
kps
Specifies that selector should query a KPS table.
.alias
Specifies the full alias of the KPS table, including the collection alias p refix if any (for example, User).
[
Indicates the start of a table property reference using a[ bracket.
key
The key value to query the table (for example, http.querystring.id). ]
Indicate the end of a table property reference using a ] bracket.
.
property
The field to retrieve from the returned row (for example, age).
}
Indicate the end of the selector using a } bracket.
You can also use a composite key, for example:
${kps.alias[key1][key2].property}
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 26
4 Access KPS data using selectors
KPS selector examples
For an example of accessing KPS data from a selector using a primary key, see Get started with KPS on page 10. For examples of selectors that use both primary and composite keys, see Configure database KPS storage on page 47. The following table shows more examples of KPS selectors: Selector
${kps.User
[http.querystring.id].firstName}
Description
l Get row from KPS table with User alias
l Use key supplied in HTTP query string (id)
l Return firstName field of row
${kps.User
["[email protected]"].age}
l Get row from KPS table with User alias
l Use constant key "[email protected]" with quotation marks
l Return age field of row
${kps.User
l Get row from KPS table with User alias
[http.querystring.firstName]
[http.querystring.lastName].email}
l Use key supplied in HTTP query string (firstName and lastName)
l Return email field of row
For more details on selectors, see the API Gateway Policy Developer Guide.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 27
Manage a KPS using
kpsadmin
5 The kpsadmin command-line tool provides KPS management functions, independent of data source. This tool is especially useful for development use. Note
In production, you should use data source-specific tools and administration procedures for data backup, restore, security, optimization, monitoring, and so on.
Start kpsadmin
From a command prompt, start kpsadmin. For example:
Windows
INSTALL_DIR\Win32\bin\kpsadmin.bat
UNIX
INSTALL_DIR/posix/bin/kpsadmin
Start in verbose mode
To run kpsadmin in verbose mode, use the -v option. kpsadmin will then show all REST messages that are exchanged with the API Gateway. This is useful for debugging. For example:
kpsadmin -v
Select kpsadmin operations
This section describes the kpsadmin operations that are available. When you first select an operation, you must enter the following:
l API Gateway group to use
l Admin API Gateway in that group that handles KPS requests
Note
This is the Admin API Gateway used for KPS purposes only, and should not be confused with the Admin Node Manager.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 28
5 Manage a KPS using kpsadmin
l KPS collection to use in the group
l KPS table to use in the collection
You can change this selection at any time. Table operations
The kpsadmin table operations are as follows:
Table operation
Description
Create Row
Create a row in the selected table.
Read Row
Read a row by primary key in the selected table.
Update Row
Update a row in the selected table. The row is specified by primary key.
Delete Row
Delete a row in the selected table. The row is specified by primary key.
List Rows
List all rows in the table.
Table administration operations
The kpsadmin operations for table administration are as follows:
Table
Administration
Description
Clear
Clear all rows in the table.
Backup
Back up the table data. The generated backup UUID is required when restoring the data.
Restore
Restore table data. The table must be empty before you restore.
Re-encrypt
Re-encrypt encrypted data in the table. Use this option when the encryption passphrase has been changed for the API Gateway group. The table will be offline after a passphrase change.
You must use this option to re-encrypt the data. You must enter the old API Gateway passphrase to proceed. Data is re-encrypted using the current API Gateway passphrase.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 29
5 Manage a KPS using kpsadmin
Table
Administration
Description
Re-create
Recreate a table. This is useful in development if you wish to change the table structure. This procedure involves dropping and recreating the table, so all existing data will be lost. The steps are as follows:
1. Back up (optional).
Backup the data if necessary using kpsadmin.
2. Deploy the correct configuration. First redeploy the correct configuration using Policy Studio. This may result in some KPS deployment errors. The changes you have made may no longer match the stored data structure.
3. Re-create the table with the correct configuration.
Select the Re-create option using kpsadmin.
4. Restore (optional) Restore the data using kpsadmin. If you have made key or index changes, the data should import directly. If you have made more extensive changes (for example, renaming fields or changing types), you must upgrade the data to match the new table structure.
Table Details
Display information about a table and its properties.
Collection administration operations
The kpsadmin operations for collection administration are as follows:
Collection
Administration
Description
Clear All
Clear all data in all tables in the collection.
Backup All
Back up all data in all tables in the collection.
Restore All
Restore all data in all tables in the collection.
Re-encrypt All
Re-encrypt all data in all tables in the collection.
Collection Details
Display information about all tables in the collection.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 30
5 Manage a KPS using kpsadmin
Cassandra administration operations
The kpsadmin operations for Cassandra administration are as follows:
Cassandra
Administration
Description
Show Configuration
Show the current configuration for the KPS-embedded storage service (Apache Cassandra).
General administration operations
The kpsadmin operations for general administration are as follows:
General
Description
Change Table
Change the currently selected table.
Change Collection
Change the currently selected collection, and select a table in that collection.
Change Group or API Gateway
Refresh the configuration, and change the currently selected API Gateway group and KPS Admin API Gateway.
Debug Mode
Enable or disable debug mode. To enable, you must enter the API Gateway group passphrase. Encrypted data in KPS tables is then shown in the clear. This can be useful for debugging issues on the API Gateway.
Example switching data source
This example shows how to switch from Cassandra storage to file storage.
Step 1: Backup collection data using
kpsadmin
To copy the current data in the collection to the new data source, back up the collection data using kpsadmin option 21) Backup All.
The backup UUID is highlighted in the following example:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 31
5 Manage a KPS using kpsadmin
Step 2: Create a new data source
To create the new data source, perform the following steps:
1. In the Policy Studio tree, select Key Property Stores > Samples.
2. Select the collection Data Sources tab. 3. Click Add > Add File at the bottom right.
4. Enter a file data source Name and Description. 5. Enter a Directory Path (for example, ${VINSTDIR/kps/samples). Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 32
5 Manage a KPS using kpsadmin
Tip
You can include ${VINSTDIR} or ${VDISTDIR} to indicate the API Gateway instance directory or install directory respectively. Make sure to use \ on Windows or / on UNIX. If the directory does not exist, it is automatically created.
6. Select the collection Properties tab. 7. Change the collection Default data source to use the new data source:
Step 3: Deploy the configuration
Click the Deploy button in the Policy Studio toolbar.
Step 4: Restore collection data using
kpsadmin
If you made a backup in step 1, to restore the collection data, perform the following steps:
1. Using kpsadmin, select option 22) Restore All. 2. Enter the backup UUID noted in step 1. For example:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 33
6 Configure Apache
Cassandra KPS storage
Apache Cassandra provides a highly available (HA) data storage option for KPS. By default, each API Gateway runs an embedded Cassandra server. You must configure two or more API Gateways to provide a HA data service. A single API Gateway, non-HA system does not require any configuration. For an overview of this configuration, see Single node, out-of-the-box, non-HA configuration on page 37.
This guide describes use cases tested with API Gateway. For production deployments, see the Apache Cassandra and Datastax documentation. You should ensure that you are familiar with Cassandra configuration and administration requirements. Note
nodetool repair must be run at regular intervals to ensure that deleted data remains deleted in a cluster. For example use cases, see Apache Cassandra operations for API Gateway on page 69. Cassandra configuration
In the API Gateway, Cassandra configuration and data are stored in the following directory:
INSTALL_DIR/groups/<group-id>/<instance-id>/conf/kps/cassandra
You can configure API Gateway to run an embedded Cassandra server and client. The Cassandra configuration is stored as follows:
File
Description
cassandra.yaml
Cassandra server configuration file.
client.yaml
Cassandra client configuration file.
jvm.xml
JMX configuration required for Cassandra administration.
commitlog
Cassandra data directory.
data
Cassandra data directory.
saved_caches
Cassandra data directory.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 34
6 Configure Apache Cassandra KPS storage
Cassandra ports
This section describes the available ports in the Cassandra configuration files:
cassandra.yaml
The Cassandra server configuration file includes the following:
Property
Description
Default
storage_port
TCP port for server–server communication. The same port must be used across the cluster.
7000
listen_address
TCP Address for server–server communication. Each server must have a unique address.
localhost
rpc_port
TCP port for client communication.
9160
rpc_address
TCP address for client communication.
localhost
seed_provider/
parameters/seeds
At least one seed must be available when bootstrapping a new node for the first time.
localhost
client.yaml
The Cassandra client configuration file includes the following:
Property
Description
Default
hosts
Comma-separated list of server address:port pairs. localhost:9160
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 35
6 Configure Apache Cassandra KPS storage
jvm.xml
The JMX configuration file includes the following:
Property
Description
Default
com.sun.management.jmxremote.port
Cassandra JMX monitoring port. After the initial handshake, the JMX protocol requires that the client reconnects on a randomly chosen port (1024+).
7199
1024+
To enable JMX, remove the <if> clause from this file.
Note
You must ensure the following:
l conf/kps/cassandra directory is included as part of your API Gateway backup plan. For more details on backup, see the API Gateway Administrator Guide.
l Changes to cassandra.yaml, client.yaml,and jvm.xml require an API Gateway restart.
l cassandra.yaml is unique to each API Gateway instance because it contains a unique listen_address for each API Gateway.
Secure ports
The following firewall rules apply for securing ports:
l storage_port:listen_address must be available to other hosts in the cluster.
l rpc_port:rpc_address and com.sun.management.jmxremote.port can be restricted to local addresses.
l Use SSH to get to the machine to run Cassandra client and administration tools such as nodetool and cassandra-cli.
Cassandra logging
Cassandra logging is set to ERROR level by default. To enable Cassandra debug output, perform the following steps:
1. Edit the following file:
INSTALL_DIR/system/lib/log4j.properties
2. Update ERROR to DEBUG for log4j.logger.org.apache.cassandra (server) and me.prettyprint.cassandra (client).
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 36
6 Configure Apache Cassandra KPS storage
3. Restart API Gateway.
Cassandra configuration steps
The main Cassandra configuration steps are outlined as follows:
1. Configure the Cassandra topology in cassandra.yaml, client.yaml, and jvm.xml.
2. Start API Gateways with embedded Cassandra servers
3. Configure the Cassandra replication factor 4. Configure request consistency levels
Step 1: Configure the Cassandra topology
You must decide on the Cassandra topology, replication factor, and consistency levels for your application. This section describes recommended and tested topology configurations in cassandra.yaml, client.yaml, andjvm.xml. In development, you can always start with one node and reconfigure later. Single node, out-of-the-box, non-HA configuration
This is the out-of-the-box API Gateway configuration. No further configuration is required. This configuration is described as follows:
l Cassandra server A is the seed node
l Server–server communication on port 7000 (must be the same across the cluster)
l Clients connect to their local server
l JMX is disabled (not needed for single node system)
l Consistent reads and writes
l No HA
The following diagram shows an example configuration: Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 37
6 Configure Apache Cassandra KPS storage
Note
When a Cassandra server starts up for the first time, it contacts a seed node to obtain information about other servers in the cluster. In a multi-server cluster, at least one seed node must be available when starting a Cassandra server for the first time.
Multiple nodes, single host, out-of-the-box, non-HA
configuration
When running multiple API Gateways on the same host, the first API Gateway to run acts as a Cassandra server. Subsequent servers to run act as clients. This is sufficient when getting started. However, the Cassandra server designation depends on the run order. Note
On API Gateways that should run as Cassandra clients, rename or remove the cassandra.yaml file, and restart these API Gateways. If a client starts up before a server, you will get multiple errors such as:
me.prettyprint.hector.api.exceptions.HectorException:All host pools marked down. Retry burden pushed out to client However, this is expected. The client will not be able to field Cassandra requests until a server is available.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 38
6 Configure Apache Cassandra KPS storage
The following diagram shows an example configuration: Note
This is a client–server configuration. If the server API Gateway is not available, KPS data and functionality will not be available to client API Gateways. If you want higher availability with Cassandra, configure a suitable HA configuration as described in the following sections.
Multiple node prerequisite: IP address required per Cassandra server
For true HA, you should use separate hosts. However, for development, you can set up a test IP address. For example, you can configure multiple IP addresses on a single host as follows: Windows
Edit or create the following file: C:\Windows\System32\drivers\etc\hosts
Add the following lines: 127.0.0.1 127.0.0.N Linux Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 39
6 Configure Apache Cassandra KPS storage
Enter the following command:
sudo ifconfig lo:N 127.0.0.N netmask 255.0.0.0 up
Where N is 2, 3, 4.
The following example shows ifconfig o utput on Linux:
Two-node HA eventual consistency
This is a two-API Gateway system, with IP addresses ipA and ipB. Note the Multiple node prerequisite: IP address required per Cassandra server on page 39. This configuration is described as follows:
l Two Cassandra servers A and B defined in a cluster (server A is the seed node) l Server–server communication on port 7000. This must be the same across the cluster
l Clients connect to their local server
l JMX is enabled for administration
The Cassandra parameter configuration includes the following:
l Eventual Consistency: Read = Write = ONE
l Replication factor: 2
l Can survive the loss of one node
l Each node holds 100% of the data
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 40
6 Configure Apache Cassandra KPS storage
The following diagram shows an example configuration: Three-node HA full consistency
This is a three-API Gateway system, with IP addresses ipA, ipB, and ipC. Note the Multiple node prerequisite: IP address required per Cassandra server on page 39. The following diagram shows an example configuration: Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 41
6 Configure Apache Cassandra KPS storage
This configuration is described as follows:
l Three servers A, B, and C defined in a cluster (server A is the seed node) l Server–server communication on port 7000 (must be the same across the cluster)
l Clients connect to their local server
l JMX enabled for administration
The Cassandra parameter configuration includes the following:
l Consistent: Read =Write = QUORUM
l Replication factor: 3
l Can survive the loss of one node
l Each node holds 100% of the data Two-node HA eventual consistency, with additional
client-only node
This is similar to the previous two node systems with an additional client-only node. For example, you would choose this option for the following reasons:
l There are resource usage issues or restrictions on certain servers in the cluster (for example, disk, memory, CPU, JARs, classpath, and ports)
l You do not want to store data at rest on certain servers
l You want to separate the data layer from the application layer
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 42
6 Configure Apache Cassandra KPS storage
Note
If either Cassandra server goes down, the client automatically fails over to the other server.
The following diagram shows an example configuration: Step 2: Start API Gateways with embedded
Cassandra servers
On first time run, perform the following steps:
1. Ensure that the seed node is started first. Wait for this node to start. 2. Start up other nodes in turn. Wait for each node to start.
Tip
You can verify the configuration by running ./kpsadmin and selecting Option 30) Show configuration.
Step 3: Configure the Cassandra replication
factor
To configure the replication factor, perform the following steps:
1. Run cassandra-cli in an API Gateway bin directory.
2. Execute the following commands:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 43
6 Configure Apache Cassandra KPS storage
./cassandra-cli -h 127.0.0.2
use kps;
update keyspace kps with strategy_
options = {replication_factor:2};
quit;
In this example, the replication factor is 2. Enter the correct replication factor for your Cassandra topology. For more details, see Step 1: Configure the Cassandra topology on page 37.
3. Synchronise and verify this change across the cluster by executing nodetool repair against each node. In the following example, nodetool is executed against a two-node cluster running on IP addresses of 127.0.0.1 and 127.0.0.2:
./nodetool -h 127.0.0.2 repair kps
./nodetool -h 127.0.0.3 repair kps
./nodetool -h 127.0.0.2 ring kps
./nodetool -h 127.0.0.3 ring kps
You should see an effective ownership on each node of 100%. For example:
Note
Cassandra 1.2.18 nodetool requires the location of cassandra.yaml. You must update apigateway/system/conf/cassandra-tools-jvm.xml to specify location of cassandra.yaml. Do not use $VINSTDIR because the system does not know which API Gateway to resolve to. For example:
<ConfigurationFragment>
<!-- for the win32 JVM, ensure that the path to the jvm.dll is available. POSIX style hosts do this using scripts. -->
<PathAdd name="PATH" value="$VDISTDIR/win32/jre/bin/server" />
<JVMSettings>
<ClassDir name="$VDISTDIR/system/lib/modules" />
<ClassDir name="$VDISTDIR/system/lib/modules/cassandra/server" />
<SystemProperty name="cassandra.config" value="file://$VDISTDIR/groups/group-2/instance1/conf/kps/cassandra/cassandra.yaml"/>
</JVMSettings>
</ConfigurationFragment>
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 44
6 Configure Apache Cassandra KPS storage
Step 4: Configure request consistency levels
In Policy Studio, set the request consistency levels specified in your Cassandra topology. For more details, see Step 1: Configure the Cassandra topology on page 37. To configure request consistency levels, perform the following steps:
1. Select a KPS collection in Policy Studio (for example, Key Property Stores > Samples). 2. Select the Data Sources tab. 3. Select the Default Embedded Data Source, and click Edit at the bottom. 4. In the dialog, select the appropriate Read Consistency Level and Read Consistency Level values for your topology.
5. Click Deploy in the Policy Studio toolbar.
The following example shows s Cassandra consistency level set to one for read and write for a collection called Samples:
Disable Cassandra storage
If you do not want to use Cassandra storage, and you are not using Cassandra for OAuth token storage, you can disable Cassandra storage. To disable Cassandra storage, perform the following steps:
1. Run the following sample script against the API Gateway. This updates the API Gateway configuration by switching any Cassandra usage to file usage. Windows
cd INSTALL_DIR\samples\scripts
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 45
6 Configure Apache Cassandra KPS storage
run.bat cassandra/disableCassandra.py
UNIX
cd INSTALL_DIR/samples/scripts
./run.sh cassandra/disableCassandra.py
2. Restart the API Gateway. 3. If you are running multiple API Gateways in the group, deploy the updated configuration to these API Gateways and restart them.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 46
7 Configure database
KPS storage
KPS data can be stored in a relational database. API Gateway supports the following databases: l Oracle l IBM DB2 l Microsoft SQL Server l MySQL
The options for database storage are as follows:
l Shared database storage: Data for multiple KPS tables is stored in a single dedicated database table. Data is encoded in JSON before being stored. This approach is very flexible because it allows maps and lists to be stored in tables. It also minimizes administration overhead because only a single database table needs to be created and managed. This is the recommended approach. For more details, see Shared database storage on page 47.
l Per-table database storage Each KPS table is backed by a single database table. Each property in the KPS table maps to a corresponding column in the database table. This approach allows existing tables to be reused, and allows more precise tuning at database level. However, it also has significant limitations (for example, supported data types, and adding and viewing data). For more details, see Per-table database storage on page 51.
Shared database storage
For shared database storage, a single database table is used to store data for multiple KPS tables. This section describes the storage configuration steps.
Step 1: Create a KPS database table
Create a database table called kps_object by executing the appropriate SQL script for your database system. SQL scripts are available in the following location:
INSTALL_DIR/system/conf/sql/DB_NAME/kps.sql
Note
For Oracle, ensure that the database is created with the AL32UTF8 character set encoding to support UTF8. Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 47
7 Configure database KPS storage
Step 2: Set up an external connection to the
database
To access this table from the API Gateway, you must setup an external database connection from the API Gateway. For more details, see the API Gateway Policy Developer Guide. The following shows example database connection settings in Policy Studio:
Note
For MySQL, the table creation script specifies UTF8. You must also use the correct JDBC connection URL. Update the connection URL field to specify unicode & UTF8. For example:
jdbc:mysql://testserver:3306/kps?useUnicode=true&characterEncoding=UTF-8
Step 3: Use the external connection in a KPS
collection
When creating a KPS collection, you can select database storage in the Default datasource field. The following example shows an SQL database selected:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 48
7 Configure database KPS storage
Alternatively, you can add a database storage option to the collection later. On the Data Sources tab, select Add > Add Database. For an example, see Step 2: Add the database data source to the KPS collection on page 52.
For an example with file storage, see Configure file-based KPS storage on page 59.
Database storage information
The following describes how KPS data is stored in a database:
l The maximum primary key length in a KPS row is 255 characters
l The maximum KPS table name length is 255 characters
l KPS rows are JSON encoded
l Optimistic locking is used and is enforced using a version column
Increase row size
You can increase the maximum KPS row size by changing the largeValue column. For example, to support image icons in MySQL, enter the following command:
alter table kps_object modify column largevalue mediumtext;
Logging for shared table storage
Shared database storage uses Apache OpenJPA to handle the communication between KPS and the back-end database. You can use OpenJPA logging to view the SQL requests transmitted to the database and their responses. This information can be useful for debugging. You can configure OpenJPA logging using Apache log4j properties.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 49
7 Configure database KPS storage
Enable OpenJPA debug logging
To enable OpenJPA debug logging:
1. Edit the following file: INSTALL_DIR/apigateway/system/lib/log4j.properties
2. Add the following settings:
log4j.rootLogger=DEBUG, A1, Vordel
# OpenJPA
log4j.category.openjpa.Tool=DEBUG
log4j.category.openjpa.Runtime=DEBUG
log4j.category.openjpa.Remote=DEBUG
log4j.category.openjpa.DataCache=DEBUG
log4j.category.openjpa.MetaData=DEBUG
log4j.category.openjpa.Enhance=DEBUG
log4j.category.openjpa.Query=DEBUG
log4j.category.openjpa.jdbc.SQL=DEBUG
log4j.category.openjpa.jdbc.SQLDiag=DEBUG
log4j.category.openjpa.jdbc.JDBC=DEBUG
log4j.category.openjpa.jdbc.Schema=DEBUG
3. Restart the API Gateway. 4. Verify that debug statements are written to the log.
Disable OpenJPA debug logging
To disable OpenJPA debug logging:
1. Edit the following file: INSTALL_DIR/apigateway/system/lib/log4j.properties
2. Substitute ERROR for DEBUG in the log4j.category.openjpa settings.
3. Restart the API Gateway. 4. Verify that no debug statements are printed to the log.
For more information on Apache OpenJPA logging, see:
http://openjpa.apache.org/builds/2.2.2/apache-openjpa/docs/main.html Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 50
7 Configure database KPS storage
Per-table database storage
You can use this option to map a KPS table to a single database table. The structure of both tables must match, so a new database table is required for each new KPS table. When the KPS table is queried, an SQL statement is executed to retrieve the correct row from the underlying database table. This SQL statement is provided by the user in Policy Studio.
However, tables that use per-table database storage have significant limitations:
l Data cannot be added through KPS, but only directly through the database
l Data cannot be viewed in kpsadmin or API Gateway Manager, but can only be read by selectors at runtime
l Tables can only contain simple data types, not maps or lists
KPS tables can be queried using simple or composite keys. This section shows examples of both.
Map a database table using a single key
In this example, a KPS table is accessed using a single key property. This key is used to retrieve the correct row from the database table. This example uses the following User table and data created using a MySQL client:
CREATE TABLE User (
email VARCHAR(100),
password VARCHAR(100),
firstName VARCHAR(100),
lastName VARCHAR(100),
age INT
);
insert into User (email, password, firstName, lastName, age) values ("[email protected]", "password", "Ralph", "Jones", 30);
insert into User (email, password, firstName, lastName, age) values ("[email protected].com", "blah", "Kathy", "Adams", 35);
Step 1: Set up an external connection to the
database
To access a database from the API Gateway, you must set up an external database connection (see Step 2: Set up an external connection to the database on page 48). This example uses the configuration from Get started with KPS on page 10. Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 51
7 Configure database KPS storage
Step 2: Add the database data source to the
KPS collection
To add a database data source, perform the following steps:
1. On the KPS collection Data Sources tab, select Add > Add Database.
2. Specify the Database Connection in the dialog (for example, Test DB Connection):
Step 3: Map the SQL table to a KPS table
From an existing KPS collection, perform the following steps:
1. Right-click the KPS collection, and select Map SQL Table:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 52
7 Configure database KPS storage
2. Enter an alias in the dialog (for example mapUser) :
3. Enter a database-specific JDBC SQL query to retrieve the required data. For example:
select * from User where email = ?
4. On the Properties tab of the new KPS table, select the new database data source in Override
the default data source with the following:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 53
7 Configure database KPS storage
Step 4: Define the KPS table structure
You must define a KPS table structure into which data will be read. You must specify the fields that you expect to read with the SQL query. In this example, all fields in the table are read using an asterisk (*) in the SQL query. This lists all fields, so the order does not matter in this case. However, the names and type must match the result returned by the SQL query.
In this SQL query, email is the primary key. You specify email as the property to use in corresponding selector queries:
For example, you can use the following selector:
${kps.mapUser
["[email protected]"].age}
Note
This syntax uses ASCII quotation marks (").
This selector generates the following SQL query:
select * from User where email = "[email protected]"
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 54
7 Configure database KPS storage
Step 5: Define the policy and path
You can reuse the policy and path from Get started with KPS on page 10 with one change—use the mapUser KPS alias for this new table. For example:
Step 6: Deploy and run
Click Deploy in the Policy Studio toolbar.
To run the policy in your browser, go to:
http://localhost:8080/[email protected] This URL specifies the user ID (email) as [email protected] For example, the result is as follows:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 55
7 Configure database KPS storage
How to map a database table using a
composite key
This section modifies the example in Map a database table using a single key on page 51. This section shows how a table with a composite secondary key can be accessed from a selector. In this version, the secondary key is {firstName, lastName}. Step 1: Modify the KPS table
You must update the database-specific JDBC SQL query in the KPS table to retrieve the required data. To modify the KPS table, perform the following steps: 1. On the Properties tab in the Query field, enter the following:
select * from User where firstName = ? and lastName = ?
For example:
2. On the Structure tab, change the selector properties to firstName,lastName:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 56
7 Configure database KPS storage
Step 2: Modify the policy
You must update the Set Message filter in your policy to use firstName and lastName parameters. For example:
Enter the following in the Message Body field, using a single line for each entry (Email, First Name, Last Name, and Age):
========================
User
===
Email: ${kps.mapUser[http.querystring.firstName]
[http.querystring.lastName].email}
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 57
7 Configure database KPS storage
First Name: ${kps.mapUser[http.querystring.firstName]
[http.querystring.lastName].firstName}
Last Name: ${kps.mapUser[http.querystring.firstName]
[http.querystring.lastName].lastName}
Age: ${kps.mapUser[http.querystring.firstName]
[http.querystring.lastName].age}
========================
Step 3: Deploy and run
Click Deploy in the Policy Studio toolbar.
To run the policy in a browser, go to the following URL: http://localhost:8080/kpsGetViaSelector?firstName=Kathy&lastName=Adams For example, the result is as follows:
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 58
Configure file-based KPS
storage
8 You can store KPS data in a directory on the file system. Each table is stored in a single JSON file. File-based storage is specified at the KPS collection or KPS table level. Note
File-based KPS storage is most suited to single API Gateway deployments. In a multiAPI Gateway scenario, file replication or a shared disk is required to ensure that all API Gateways use the same data.
File-based KPS tables are read and cached by API Gateways when they start up. If data is modified, all API Gateways must be restarted to pick up the changes.
Configure a file-based KPS collection
You can configure a file-based KPS data source when creating a KPS collection, or add one later on the Data Source tab. For example, the following settings are available when editing file-based collection:
These settings are described as follows:
Name
Description
Name
Collection-unique data source name.
Description
Optional description.
Directory
Path
The directory name where table data for the collection is stored. If the directory does not exist, it is automatically created. If this directory is not specified, the directory path defaults to ${VINSTDIR}/conf/kps.
The path can include VDISTDIR or VINSTDIR variables. These are resolved to the API Gateway instance and installation directories. For example, ${VDISTDIR}/mydata/samples. Remember to use the correct path separator (/ on UNIX and \ on Windows).
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 59
Appendix A: KPS FAQ
This appendix answers the frequently asked questions on KPS and API Gateway, KPS storage, and Apache Cassandra.
KPS and API Gateway
This section includes frequently asked questions on KPS and API Gateway:
What is KPS used for in API Gateway?
In addition to use in your API Gateway policies, KPS is used as an option for OAuth token storage, and Client Application Registry. If you do not wish to use these features, or do not wish to use the embedded Apache Cassandra service as a storage mechanism for these features, see Configure Apache Cassandra KPS storage on page 34 for details on how to disable Cassandra storage.
What is KPS not suitable for?
KPS is not suitable for complicated data models, ad hoc queries, or where full ACID transaction support is required. KPS does not enforce referential integrity.
What are the transaction semantics of KPS?
Individual KPS operations are atomic (A), isolated (I), and durable (D). Consistency (C) depends on the data storage mechanism chosen and the number of API Gateways in a group. With file storage, data is consistent in a single API Gateway. With supported database storage, data is consistent. Cassandra storage allows consistency levels to be set per KPS table. KPS does not provide transactions across multiple operations. You cannot issue a set of KPS operations and roll them back.
What is the KPS collection alias prefix for?
This provides an optional namespace for a KPS collection to help ensure that tables in the collection have a unique alias. In most cases, you can leave this prefix empty. Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 60
Appendix A: KPS FAQ
Why are some collections hidden? How can
you show them?
Some out-of-the-box KPS collections are hidden in Policy Studio to prevent accidental modification. To show hidden KPS collections, perform the following steps:
1. Edit the following file:
INSTALL_DIR/policy_studio/policystudio.ini
2. Set the show.internal.kps.collection system property to true.For example:
3. Restart Policy Studio. Note
You will need to edit these hidden collections if you wish to change Cassandra consistency levels (for example, when switching to an HA setup).
How do I change API Gateway group
passphrase?
To change the group passphrase, perform the following steps:
1. Change the group passphrase in Policy Studio. For details, see the API Gateway Administrator Guide. 2. In kpsadmin, select the Collection Administration, Re-encrypt All option to reencrypt the data in each collection. 3. You will be asked to enter the old API Gateway passphrase. This passphrase is used to decrypt the data. The data is then re-encrypted with the current API Gateway passphrase.
For more details, see Manage a KPS using kpsadmin on page 28. Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 61
Appendix A: KPS FAQ
KPS storage options
This section includes frequently asked questions on KPS storage mechanisms:
Is Apache Cassandra storage required? Can
you use file or database?
Apache Cassandra is the default out-of-the-box storage, but is not a requirement. You can switch to file or database storage. For more details, see How do you switch storage for a KPS collection? on page 62.
How do you switch storage for a KPS
collection?
The general steps to switch storage for a KPS collection are as follows:
1. Back up the data for a collection using kpsadmin. 2. Add and configure a new data source (file, database, or Cassandra) using Policy Studio. 3. Deploy the configuration in Policy Studio. 4. Restore the data using kpsadmin. For more details, see the example in Manage a KPS using kpsadmin on page 28. Why use database storage?
You may already have a relational database system and expertise that you wish to use. For more details, see Configure database KPS storage on page 47.
Why use file storage?
File storage is very easy to use. Data is written to very simple JSON encoded files. File storage is suitable for single API Gateway use in development and production. If you have more than one API Gateway sharing the files, you must restart these API Gateways after a KPS update. It is only suitable here for rarely changing, read-mostly data. For more details, see Configure file-based KPS storage on page 59.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 62
Appendix A: KPS FAQ
When can you use kpsadmin? When should
you use storage-specific tools?
You can use kpsadmin anytime. However, it is especially useful for development use. In production, you should use storage-specific tools and procedures (for example, for data backup). For more details on kpsadmin, see Manage a KPS using kpsadmin on page 28. Apache Cassandra
This section includes frequently asked questions on the Apache Cassandra database:
Why use Cassandra as a KPS storage
option?
Cassandra is a key-value store. Cassandra has a non-restrictive Apache 2.0 license. It provides HA, is embeddable in a Java application (API Gateway), and has an active community.
What version of Cassandra does the
API Gateway use?
The API Gateway ships with a version of Cassandra 1.2.18. Non-required JARs and scripts have been removed. The cassandra-cli and nodetool utilities are included. What does all host polls marked down
mean?
This means that the embedded client cannot connect to the embedded Cassandra server. For details on how to resolve this issue, see Troubleshoot KPS error messages on page 65.
Can you use an external Cassandra
instance?
You can modify the client.yaml file in API Gateway Cassandra configuration to point to an external Cassandra instance. This will reduce resource usage requirements in the API Gateway process.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 63
Appendix A: KPS FAQ
How do you set Cassandra consistency
levels?
For details, see Configure Apache Cassandra KPS storage on page 34.
How do you disable Cassandra?
For details, see Configure Apache Cassandra KPS storage on page 34.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 64
Appendix B: Troubleshoot KPS
error messages
This appendix describes various KPS error messages, and explains how to resolve them. All platforms
This section explains how to troubleshoot KPS errors on all platforms:
All host polls marked down
This error means that the embedded client cannot connect to the embedded Cassandra server. To resolve this issue, perform the following steps:
l Check that all ports and addresses are correct in cassandra.yaml to verify that the endpoints are what you expect. l Run kpsdamin, and select Cassandra Administration, Show Configuration to read this configuration. And verify the result is what you expect. For more details, see Manage a KPS using kpsadmin on page 28.
l Enable Cassandra debug logging. For more details on Cassandra configuration and logging, see Configure Apache Cassandra KPS storage on page 34.
Nodetool reports failed to connect
When running the nodetool command against a remote host in the Cassandra cluster, the following error is output:
Failed to connect to 'REMOTE_IP:7199: Connection refused'
To resolve this issue, perform the following steps: 1. On each node, edit the jvm.xml file in the following directory: INSTALL_DIR/apigateway/groups/GROUP_ID/INSTANCE_ID/conf/kps/cassandra
2. Add the following JVM argument:
java.rmi.server.hostname=BIND_ADDRESS
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 65
Appendix B: Troubleshoot KPS error messages
For more details on Cassandra configuration, see Configure Apache Cassandra KPS storage on page 34.
May not be enough replicas present to
handle consistency level
For example, this error is received in a two-node Cassandra system with an eventual consistency of read = write = ONE. To resolve this issue, you must ensure that the replication is correctly. For example, you can use the following nodetool command:
./nodetool -h 127.0.0.2 ring kps
Datacenter: datacenter1
==========
Replicas: 1 xxxxx
Address Rack Status State Load Owns Token 7564491331177403445
127.0.0.2 rack1 Up Normal 1.19MB 35.10% 1090016113642762867
127.0.0.3 rack1 Up Normal 2.17MB 64.90% 7564491331177403445
Note
In this case, the number of replicas is 1 and Owns is not 100% per node. You should set the replication factor, synchronize the changes, and verify as described in Configure Apache Cassandra KPS storage on page 34
Windows only
This section explains how to troubleshoot KPS errors on Windows only:
Node running in client mode
On Windows, the software detects that a service is already running on the local machine, even though there is only one API Gateway running. To resolve this issue, perform the following steps:
1. Edit cassandra.yaml and client.yaml. 2. Change localhost to IP name or address. 3. Restart API Gateway.
For more details on Cassandra configuration, see Configure Apache Cassandra KPS storage on page 34.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 66
Appendix B: Troubleshoot KPS error messages
UTF-8 characters not displaying correctly in
kpsadmin
This is a known limitation of the Windows command console. To view UTF-8 data correctly, run kpsadmin in a Cygwin console instead. For more details on kpsadmin, see Manage a KPS using kpsadmin on page 28.
FSUTIL utility bug
On Windows, the following error may occur:
The FSUTIL utility requires a local NTFS volume. at org.apache.cassandra.utils.FBUtilities.exec(FBUtilities.java:573) This is an Windows XP issue, which also applies to NTFS (not just FATxx). For more details, see:
http://support.microsoft.com/kb/322275
Path length issue >255 characters
On Windows, you may see an error such as the following:
java.io.IOException: Exception while executing the command: cmd /c mklink...
command error Code: 1, command output: The system cannot find the path specified.
To resolve this issue, perform the following steps:
1. Use a short path for Cassandra data. 2. Update the cassandra.yaml data directories:
data_file_directories: [dir-x]
commitlog_directory: dir-x
saved_caches_directory: dir-x
3. Restart the API Gateway.
For more details on Cassandra configuration, see Configure Apache Cassandra KPS storage on page 34.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 67
Appendix B: Troubleshoot KPS error messages
Could not drop kps keyspace using
cassandra-cli
On Windows, you may see this error which is caused by Apache thrift.
To resolve this issue, you should drop individual column families and the kps_schema record. For example, for a samples_user table, paste the following into cassandra-cli: use kps;
assume kps_schema validator as utf8;
assume kps_schema comparator as utf8;
assume kps_schema keys as utf8;
assume samples_user validator as utf8;
assume samples_user comparator as utf8;
assume samples_user keys as utf8;
drop column family samples_user;
del kps_schema['samples_user'];
Note
The assume commands must be executed once.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 68
Appendix C: Apache Cassandra
operations for API Gateway
This appendix describes important Apache Cassandra operations for node repair, and backup and recovery.
nodetool repair
The nodetool repair command repairs inconsistencies across all Cassandra replicas for a given range of data. For more details, see:
http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_repair_
nodes_c.html
You should execute this command weekly, at off-peak times, and stagger execution on different nodes.
The following is a simple crontab command that executes repair every 10 minutes: [email protected]:~$ crontab */10 * * * * /home/vagrant/from/apigateway/posix/bin/nodetool -h node-3 repair kps >> /home/vagrant/nodetool.log
Note
This is a simple example from a development test. In production, do not execute repair every 10 minutes.
Backup and recovery
This section explains how to back up and restore all KPS data.
Back up and restore Cassandra node data
To back up and restore Cassandra data (online and HA), use the following instructions:
http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_backup_
restore_c.html
Note
When restoring from a snapshot, you should follow the node restart method.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 69
Appendix C: Apache Cassandra operations for API Gateway
Back up API Gateway KPS configuration
KPS configuration is stored in API Gatewaythe group configuration in the following directory:
INSTALL_DIR/apigateway/group-n/conf
You should regularly back up this directory.
Back up API Gateway Cassandra
configuration and data
API Gateway group configuration and data is stored in the following directory:
INSTALL_DIR/apigateway/group-n/instance-m/conf/kps/cassandra
This contains the Cassandra configuration (cassandra.yaml, client.yaml, and jvm.xml files). It also includes the Cassandra runtime data in the data, saved_caches, and commitlog subdirectories.
You should regularly back up this directory.
Note
For more details on API Gateway backup and disaster recovery, see the API Gateway Administrator Guide.
Replace dead node
The procedure for a standard Cassandra installation is as follows:
http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_replace_
node_t.html The procedure for API Gateway is as follows:
1. Confirm that the node is dead using nodetool status (see the Cassandra procedure above).
2. Note the address of the dead node (this is used in the last step).
3. Install a new API Gateway, or restore a backup of an existing API Gateway. Do not start the API Gateway.
4. The API Gateway Cassandra directory is: INSTALL_DIR/apigateway/group-n/instance-m/conf/kps/cassandra
Copy cassandra.yaml, client.yaml and jvm.xml from a backup or recreate in the API Gateway Cassandra directory.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 70
Appendix C: Apache Cassandra operations for API Gateway
5. Backup and remove the data, saved_caches, and commitlog d irectories if they exist. cassandra-topology.properties is required for multi-datacentre configuration. Oracle currently do not test or support this kind of configuration. For more details on the cassandra.yaml file, see the standard Cassandra procedure above. Note
auto_bootstrap should not be listed or should be set to false
6. Start the API Gateway with the -Dcassandra.replace_address option. Ensure JMX is enabled. For example: ./startinstance
-g TeamA
-n api2
-Dcassandra.replace_address=192.168.99.12
-DenableJMX
7. Remember to remove the -Dcassandra.replace_address option for subsequent starts.
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 71
Further information
This topic shows where to find more details on KPS-related information:
Feature
Documentation
KPS REST
API
The KPS REST API is documented in the following directory:
Apache
Cassandra
For details on Cassandra version 1.2.x, see the following:
INSTALL_
DIR/apigateway/docs/restapi/APIGatewayAPI/api/rest/kps.htm
l
l http://cassandra.apache.org/
l http://www.datastax.com/documentation/cassandra/1.2
Apache
Cassandra
parameter
configuration
For details on Cassandra parameter configuration, see the following:
API Gateway
database
connections
For details on how to configure database connections, see the API Gateway Policy Developer Guide. API Gateway
caches
For details on how to configure local caches, see the API Gateway Policy Developer Guide.
http://www.ecyrd.com/cassandracalculator/
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 72
Glossary
Apache Cassandra
An open source distributed database, designed to provide high availability by distributing data across multiple servers.
KPS
Key Property Store. Data management component in the API Gateway.
KPS alias
An alternative name for a KPS table that is unique in an API Gateway group.
KPS collection
A collection of related KPS tables.
KPS table
A user defined table that is managed by the KPS in the API Gateway.
Selector
A special syntax that enables API Gateway configuration settings to be evaluated and expanded at runtime based on metadata values (for example, from a KPS, message attribute, or environment variable).
Oracle API Gateway 11.1.2.4.0
Key Property Store User Guide 73
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement