Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
• When you create an authorization for another user using INVOKER TRUSTED,
user_dbname must be specified. Specify the username associated with the session user who will be sending requests to the foreign server. If you fail to specify user_dbname, the authorization will be stored in your user database.
• The authorization takes up no space in the database used to store it.
• If your credentials change on the foreign server's target platform, you must remember to replace the credentials in your authorization object. If you fail to update the invalid information, the next time that you try to reference the foreign server object, you get an error message.
• If you drop an authorization object, keep in mind that it may be used by multiple foreign server objects. You should either drop the foreign server objects or alter them so that they specify a valid authorization object. If you fail to update the invalid information, the next time that you try to reference the foreign server object, you get an error message.
Examples of Creating and Replacing the Authorization
If you plan to use the authorization to authenticate to LDAP or Kerberos on a foreign server then you must use either INVOKER TRUSTED or DEFINER TRUSTED.
The following two examples establish authorization for the user who invokes the object. The credentials are encrypted and stored as a database object in the user database.
CREATE AUTHORIZATION sales AS INVOKER TRUSTED
USER 'johnson'
PASSWORD 'Secret' ;
REPLACE AUTHORIZATION sales AS TRUSTED
USER 'williams'
PASSWORD 'topsecret' ;
If you want to make the authorization available globally, create the authorization on
TD_SERVER_DB using the DEFINER TRUSTED type.
CREATE AUTHORIZATION TD_SERVER_DB.remote_system1
AS DEFINER TRUSTED USER 'proxy_1'
PASSWORD 'Global' ;
If you use DEFINER TRUSTED, as in this example, then the credentials for johnson
are stored in the sales
authorization created in the TD_SERVER_DB database.
CREATE AUTHORIZATION TD_SERVER_DB.sales AS DEFINER TRUSTED
USER 'johnson'
PASSWORD 'Secret';
CREATE FOREIGN SERVER
Purpose
Creates a foreign server object and associates table operators with it.
Teradata QueryGrid: Teradata Database-to-Hadoop User Guide 23
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
When you create a server object, you can customize it based on its purpose. You can define multiple server objects for the same remote database, each with different characteristics needed by different users.
You can use name value pairs to define the characteristics of the foreign server. You can use global parameters to define the foreign server as a whole. Some table operators have local parameters that are specific to the operator. Some parameters can be used so that the server object overrides user selection; for example, you limit access to data by setting the table name.
Syntax operator option
table_operator
using option
database_name.
Syntax Elements
server_name
The name given to the foreign server object.
EXTERNAL SECURITY
Associates an authorization object with the foreign server. The authorization stores the encrypted credentials for a user as a database object. The Teradata QueryGrid connector passes the credentials in the authorization to the remote platform identified by the foreign server when the foreign server is accessed.
You must use EXTERNAL SECURITY TRUSTED for Teradata QueryGrid: Teradata
Database-to-Hadoop when the Hadoop platform is protected by an external security system, such as Kerberos, for example.
INVOKER
24 Teradata QueryGrid: Teradata Database-to-Hadoop User Guide
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
DEFINER
INVOKER is a keyword that indicates that the associated authorization must be present in the user database at the time that the foreign server is accessed.
Note: The user database is the database that was created for the user in the Teradata system when the user account was created.
DEFINER is a keyword that indicates that the associated authorization must be present in the database that contains the foreign server when the foreign server is accessed.
Note: The DEFAULT keyword that can be used with DEFINER in CREATE
AUTHORIZATION and REPLACE AUTHORIZATION statements is not needed in association with a foreign server.
You must use either INVOKER TRUSTED or DEFINER TRUSTED if the remote platform uses an external security system (such as Kerberos, for example) for authentication.
TRUSTED
A keyword that indicates the associated authorization object was created as TRUSTED.
authorization_name
Specifies the name of the authorization object to be used when the foreign server is accessed.
Using Option
USING
USING introduces the global name value pairs (NVPs) that provide the server definition information. USING must be followed by at least one name value pair of the form
name('value' ), but an empty value of ' ' is supported. You can create a foreign server without a USING clause, but users cannot query a foreign server until you complete the server definition with an import operator and an export operator.
The USING clause that appears in the server area (in front of the table operators ) contains global parameters that define the connection to the remote platform and can be applied to both the import and export table operators. The USING clause that appears in the operator option part of the syntax diagram contains local parameters that are used just for that table operator. A name value pair can be used in any USING clause location unless otherwise indicated as "Server only," "Import only," and "Export only."
name('value')
The name value pair or pairs that you specify to define the foreign server.
For descriptions of the name value pairs used with the server object, see "Required Name
Value Pairs" and "Optional Name Value Pairs."
Required Name Value Pairs
These name value pairs are required to create a functioning foreign server object. Additional optional name value pairs may be required to create a foreign server for a specific implementation.
hosttype
Server only.
For Teradata QueryGrid: Teradata Database-to-Hadoop, this is
('hadoop')
.
Teradata QueryGrid: Teradata Database-to-Hadoop User Guide 25
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
port
Server only. The server port number for the Hive Metastore; typically this is 9083.
server
Server only. The DNS host name or IP address for the Apache Hive Metastore
(hive.metastore.uris). You can use an application, such as Ambari, to obtain this value.
Optional Name Value Pairs
These name value pairs are optional. However, a particular server implementation may require you to define some of these name value pairs. For example, a foreign server must be defined with a Hive port to support queries that access Hive Server2.
clustername
Required when using security('kerberos')
. Specifies the directory name that stores the JAR file that contains the configuration files (core-site.xml, hdfs-site.xml, hive-site.xml, hive-site.xml, and yarn-site.xml) for the Hadoop cluster to be accessed.
This directory was set up during the Hadoop client installation on the Teradata nodes.
For example, you would use the name value pair clustername('yourcluster')
if the files are stored as follows:
• yourcluster/
• yourcluster/core-site.xml
• yourcluster/hdfs-site.xml
• yourcluster/hive-site.xml
• yourcluster/hive-site.xml
• yourcluster/yarn-site.xml
compression_codec
Export only. Specifies the type of compression to use for the exported data. The default is no compression.
The supported compression types are based on the compression codecs configured on the Hadoop system.
Note: Snappy is supported but you must specify it in hadoop_properties using
<orc.compression=SNAPPY> as the argument. Snappy is supported only for the
ORCFile file type.
You must specify the full name for the compression codec as follows:
• org.apache.hadoop.io.compress.DefaultCodec
• org.apache.hadoop.io.compress.GzipCodec
• org.apache.hadoop.io.compress.BZip2Codec
dbname
The name of the user’s database.
This parameter is optional. You can specify a dbname value in the foreign server to limit its scope to a specific database. The value specified in the USING clause in the CREATE
FOREIGN SERVER syntax overrides any corresponding value specified directly in the user query.
default_string_size
Size at which data imported from or exported to Hadoop String columns is truncated.
When applied to the import operator, the value represents the maximum number of
Unicode characters to import, and defaults to 2048 characters. When applied to the
26 Teradata QueryGrid: Teradata Database-to-Hadoop User Guide
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER export operator, the value represents the maximum number of bytes to export, and defaults to 4096 bytes. Teradata QueryGrid silently truncates the String columns at the default value set in default_string_size.
hadoop_properties ('<property1=value>, <property3=value1,value2>')
Sets specific properties used to interact with Hadoop. If there are multiple arguments, you must delimit them with angle brackets. If there is only one argument, you can omit the angle brackets. For example, the syntax for the hadoop_properties clause for a High
Availability (HA) target supports an updated syntax where multiple values can be included. In this case the properties must be enclosed by left and right angle brackets.
No spaces are allowed within or between arguments.
The High Availability hadoop properties are defined based on the name service that is defined on your hadoop server. For example, if you have the following Hadoop properties: hadoop_properties('
<dfs.client.use.datanode.hostname=true>
,<dfs.datanode.usedatanode.hostname=true>
,<dfs.nameservices=MYCOMPANY_HADOOP02>
,<dfs.ha.namenodes.MYCOMPANY_HADOOP02=nn1,nn2>
,<dfs.namenode.rpc-address.MYCOMPANY_HADOOP02.nn1=hdp230-2:8020>
,<dfs.namenode.rpc-address.MYCOMPANY_HADOOP02.nn2=hdp230-3:8020>
,<dfs.client.failover.proxy.provider.MYCOMPANY_HADOOP02=org.apache.
hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider>')
In this example, you would make the following replacements:
• Replace
MYCOMPANY_HADOOP02
with your own name service ID.
• Replace hdp230-2
and hdp230-3
with your own namenode hostnames.
It may also be necessary for you to replace nn1
and nn2
in the example above with your own namenode aliases. To verify, check the following property in your hdfssite.xml
file:
<property>
<name>dfs.ha.namenodes.MYCOMPANY_HADOOP02</name>
<value>namenode10,namenode66</value>
</property>
In this case, you would make the following replacements:
• Replace nn1
with namenode10
• Replace nn2
with namenode66
In most cases, you should set the dfs.client.use.datanode.hostname property to true. If you have a setup where your TPA nodes are on one BYNET, the Hadoop cluster is on another BYNET, and they are communicating with one another via Ethernet, then you should also set the dfs.datanode.usedatanode.hostname property to true.
hiveserver
The DNS host name or IP address of Hive Server2. This is used when a query results in the use of the HCTAS and HDROP procedures or FOREIGN TABLE SELECT in Hive.
(You can use an application, such as Ambari, to obtain this value.) If no value is specified for hiveserver then the value for server is used.
hiveport
The port for access to the Hive Server2; typically this is 10000. You can use an application, such as Ambari, to obtain this value.
Teradata QueryGrid: Teradata Database-to-Hadoop User Guide 27
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
28
merge_hdfs_files
Export only. Indicates that files under the same partition should be merged whenever possible. The default is to not merge. A value of TRUE means that files will be merged.
row_count_report_freq
The frequency with which byte count is updated in DBQL. The default is every 100 rows. You can adjust this to a larger value if the update frequency is too resource intensive.
security
Specifies the name of the external security system used for authentication on the
Hadoop cluster. This parameter is required when an external security system is in use.
The default is no security. Valid values are:
• kerberos
• ldap
Note: Teradata QueryGrid: Teradata Database-to-Hadoop supports only Kerberos authentication when used with Cloudera CDH 5.4; the use of LDAP on Cloudera CDH
5.4 is not supported.
tablename
The name of the table to be imported or exported.
This parameter is optional. You can specify a tablename value in the foreign server to limit its scope to a specific table. The value specified in the USING clause in the
CREATE FOREIGN SERVER syntax overrides any corresponding value specified directly in the user query.
temp_dbname
Import only. The value is the name of the Hadoop database to use to store temporary
Hive staging tables.
This parameter is optional. You should consider using temp_dbname when planning to use FOREIGN TABLE syntax on a foreign server or when a foreign server is set up to use usenativequalification. If no database is specified with temp_dbname, then the default Hadoop database is used.
To use the specified database, it must exist when the foreign server is created or altered to use the database. The session user must have create and write permission to the database. If multiple users use the same foreign server then the Hadoop administrator may want to consider setting up Hive authorization in such a way that temporary tables cannot be read by another user.
transformformatting
Import only. When set to 'true' it indicates that an array list data is formatted appropriately, so that it can be cast directly into a Teradata array column type based on the appropriate data type.
This parameter is optional. The value specified in the USING clause in the CREATE
FOREIGN SERVER syntax overrides any corresponding value specified directly in the user query.
updatestatistics
Export only. Indicates that the LOAD_TO_HCATALOG_abcn_n_n operator updates the table statistics after all the data has been loaded into the target Hive table. Valid values are 'true' and 'false'. A value of true means that the table statistics are updated.
Note: You must also have set hive.stats.autogather to true in your hive-site.xml file for updatestatistics to work properly.
Teradata QueryGrid: Teradata Database-to-Hadoop User Guide
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
usenativequalification
Import only. A value of 'true' indicates that SELECT queries should be pushed down to
Hive as much as possible. When a foreign server uses usenativequalification, Teradata
Database examines the following conditions:
• Hive table data size is large and there are qualifying predicates on non-partitioned
Hive columns. Large is defined as having a number of splits that is larger than the number of Teradata Database nodes.
• The queried Hive object is a view.
When either of the two conditions are met, Teradata Database constructs a Hive query from the Hive object name, the referenced columns, and the qualifying predicates. It then creates a Hive staging table (in the database specified by temp_dbname) from the constructed query and retrieves data in the staging table. The staging table is dropped after all data has been retrieved.
Valid values are 'true' and 'false.'
For queries that involve joins between two HCatalog tables, Teradata Database brings the data into Teradata spool and joins them in the database. The join is not pushed into
Hive. For example, the join syntax in the following query requires the manual
FOREIGN TABLE SELECT to be accomplished in Hive:
SELECT h1.c1, h2.c2 FROM h1@hadoop1, h2@hadoop1 WHERE h1.id = h2.id ;
username
The name of the Hadoop user's credential. This option is ignored when the security name value pair is defined for the foreign server. If no username value and no security value are defined, the foreign server uses the name of user making the request. (This is the Teradata Database user name in capital letters.)
No password is associated with a Hadoop user. HDFS and Hive check for a user name for access. If no user name is specified, then foreign server supplies the name of the session user. If HDFS and Hive are not configured for file permissions, then the user name is optional.
DO IMPORT WITH
Associates an IMPORT table operator with a foreign server.
Note: You can specify table operators in any order.
DO EXPORT WITH
Associates an EXPORT table operator with a foreign server.
Note: You can specify table operators in any order.
Operator Option
database_name.
The name of the database that contains the operator that you want to call. For example,
SYSLIB.
table_operator
The name of the table operator to use for import or export. The Teradata-to-Hadoop connector provides the following table operators for use:
Teradata QueryGrid: Teradata Database-to-Hadoop User Guide 29
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
Table Operators Connector Releases That Provide These
Operators
15.0, 15.0.1, 15.0.2, and 15.0.3
LOAD_FROM_HCATALOG_HDP1_3_2
LOAD_TO_HCATALOG_HDP1_3_2
LOAD_FROM_HCATALOG_HDP2_1_2
LOAD_TO_HCATALOG_HDP2_1_2
LOAD_FROM_HCATALOG_HDP2_3_0
LOAD_TO_HCATALOG_HDP2_3_0
LOAD_FROM_HCATALOG_CDH5_4_3
LOAD_TO_HCATALOG_CDH5_4_3
15.0.1, 15.0.2, 15.0.3, and 15.0.4
15.0.4
15.0.4
DO IMPORT WITH uses the LOAD_FROM_HCATALOG_a bcn_n_n table operators.
These table operators retrieve data from a Hadoop distributed database into Teradata
Database, where the data can be placed in tables or joined with existing tables. These table operators produce a spooled table that contains rows and columns of data from a user-specified Hadoop table that is defined in the HCatalog of the remote system.
DO EXPORT WITH uses LOAD_TO_HCATALOG_abcn_n_n table operators.
These table operators export data from Teradata Database into a Hadoop distributed database, where the data can be placed in tables or joined with existing tables.
Note: When you create a foreign server, specify the table operator name with the distribution acronym and version number that corresponds with the version of the
Hadoop distribution that you are using. For example,
LOAD_FROM_HCATALOG_CDH5_4_3 is compatible with Cloudera CDH version
5.4.3.
Supported Data Types, HCatalog File Types, and Compression
The following table shows the data types supported by the Teradata-to-Hadoop connector and how they are mapped during import and export.
Hadoop Data Type
String
Boolean
Integer
BigInt
Float
Double
BINARY
MAP
Struct
ARRAY
Teradata Database Data Type
VARCHAR UNICODE CHARSET
BYTEINT
INT
BIGINT
FLOAT
FLOAT
VARBYTE
VARCHAR UNICODE CHARSET
VARCHAR UNICODE CHARSET
VARCHAR UNICODE CHARSET
30 Teradata QueryGrid: Teradata Database-to-Hadoop User Guide
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
Hadoop Data Type
TINYINT
SMALLINT
Date
Timestamp
Decimal
VARCHAR
CHAR
Teradata Database Data Type
BYTEINT
SMALLINT
DATE
TIMESTAMP
DECIMAL
VARCHAR
CHAR
Data Import Notes
The Hadoop String data type does not have a maximum length. Teradata Database has a row size limit of approximately 64K. Exceeding this row size limit results in an error. The default
VARCHAR string size of 4096 bytes (2048 Unicode characters) permits approximately 14
Hadoop String columns to be imported successfully and held in a single Teradata Database row. (Note that the row header and columns of other types are part of the row size and may reduce the number of String columns that can be imported.)
You may want to change the default_string_size in the Foreign Server to a different value based on typical string size and number of Hadoop String columns to be imported. For example, if your row size typically exceeds the size limit you may want to take a best-fit approach and set the default_string_size to a value smaller than 2048 so that imports can be performed without error. The strings are truncated at the value set in default_string_size.
You may alternately want to import large Hadoop STRING columns using a more customized approach. For more information, see
RETURNS Clause
.
The Teradata Database data type JSON must first be cast to VARCHAR or CLOB for export.
It is imported as a VARCHAR or CLOB and cast to JSON.
BOOLEAN data types are stored as true/false literal values in Hadoop HiveQL. In Teradata,
Hadoop BOOLEAN data types are mapped to BYTEINT data types. When Teradata imports
Boolean data from Hadoop, the true/false literal values are stored as 1/0 values, respectively, in the corresponding Teradata BYTEINT column.
Data Export Notes
The Date and Timestamp are assumed to be UTC. For more information, see
Timestamp
Data .
You should be aware of the display differences that occur when you export BOOLEAN data from Teradata into Hadoop:
• If you export data into a Hive Table that you created by using the Hive command line interface, the BYTEINT data type in Teradata is mapped to a BOOLEAN in HCatalog, and the 1/0 values are displayed as true/false literals in Hive.
• If you export data into a new Hive table that you created by using the Teradata HCTAS stored procedure, then the BYTEINT data type in Teradata is mapped to TINYINT in
HCatalog, which has the same range as BYTEINT. So, the BYTEINT 1/0 values
(Boolean) are exported as-is into the TINYINT column in HCatalog.
Teradata QueryGrid: Teradata Database-to-Hadoop User Guide 31
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
In Hive, if you want to display Boolean true/false literal values in an HCatalog table, instead of the 1/0 values, you can use the built-in CAST conversion function in Hive to convert the display of a TINYINT value to the BOOLEAN primitive type. For example:
SELECT CAST(column_bool_1 AS BOOLEAN) from HCTAS_TBL; where column_bool_1 is defined in the HCatalog table as follows: column_bool_1 TINYINT,
HCatalog File Types and Compression
The following table shows the file types and compression types supported by the Teradata-to-
Hadoop connector.
Distribution Supported Import File Types Compression Supported By
File Type
Hortonworks HDP 2.1.2 and
2.3.0
Cloudera 5.4.3
• TextFile
• RCFile
• ORCFile
• TextFile
• RCFile
Compression types supported for use with TextFile:
• DefaultCodec
• BZip2Codec
• GzipCodec
Compression types supported for use with RCFile:
• Block compression
Compression types supported for use with ORCFile:
• Block compression
• SnappyCodec
Compression types supported for use with TextFile:
• DefaultCodec
• BZip2Codec
• GzipCodec
Compression types supported for use with RCFile:
• Block compression
Required Privileges
You must have CREATE SERVER privilege on the TD_SERVER_DB database to define a foreign server object. If you are associating the server with table operators, you must also have EXECUTE FUNCTION and SELECT privileges on the specified table operators.
Usage Notes
32
• The target platform of the foreign server object must be running and reachable when you create the foreign server object for it in Teradata Database.
Teradata QueryGrid: Teradata Database-to-Hadoop User Guide
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
• You can create multiple named foreign server objects that reference the same server using the same IP and port numbers.
• Foreign server object names that are stored in TD_SERVER_DB must be unique.
• Teradata treats the hosttype name value pair as special. If you specify this name value pair, you must use it in the server-area name value list.
• Name value pairs in the server area of the syntax apply to the connection to the remote platform and to both of the table operators specified in the IMPORT WITH and
EXPORT WITH clauses.
• Name value pairs in the IMPORT WITH or EXPORT WITH clause apply only to the table operator specified in the clause.
• Server options, names, and name value pairs can appear only once in the CREATE
FOREIGN SERVER syntax.
Name value pairs used within the IMPORT WITH and EXPORT WITH clauses cannot duplicate those used in the server-area name value list.
• The order of the DO IMPORT WITH and DO EXPORT WITH clauses in the CREATE
SERVER syntax does not matter.
• You must grant SELECT, INSERT, and SHOW privileges on foreign server objects to users who need to query foreign server objects.
• The use of the EXTERNAL SECURITY clause is required when the foreign server's target platform uses LDAP or Kerberos for authentication. For more information, see
LDAP and Kerberos Authentication Security .
You cannot use the following names in the name value pairs in CREATE SERVER statements:
• Columns
• hExplain
• IsNested
• Servermode
Examples of Using CREATE FOREIGN SERVER
A standard foreign server definition must contain the following NVPs:
• server
• port
• hosttype
Most foreign server definitions also use the following NVPs:
• hiveserver
• hiveport
• username hadoop_properties may also need to be defined in the following situations:
• Data Nodes have a private network (multi-homed)
• High Availability is enabled
For a description of the name value pairs used to define the foreign server, see Using Option
.
Teradata QueryGrid: Teradata Database-to-Hadoop User Guide 33
Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop
CREATE FOREIGN SERVER
Example: Typical Server Definition With IMPORT and EXPORT Table
Operators
The following example creates a server object and associates an IMPORT table operator and an EXPORT table operator with it:
CREATE FOREIGN SERVER hadoop1
USING
hosttype('hadoop')
server('192.0.2.3')
port('9083')
hiveport ('10000')
username('hive')
DO IMPORT WITH SYSLIB.LOAD_FROM_HCATALOG_HDP2_3_0
DO EXPORT WITH SYSLIB.LOAD_TO_HCATALOG_HDP2_3_0;
Example: Creating a Server Object for LDAP
The following example creates a server object for an LDAP-protected Hadoop cluster. It uses an authentication object named auth_hdp
that is located in the user database for the session user:
CREATE FOREIGN SERVER TD_SERVER_DB.hadoop2
EXTERNAL SECURITY INVOKER TRUSTED auth_hdp
USING
hosttype('hadoop')
server('hserver_name.example')
port('9083')
hiveport ('10000')
security('ldap')
DO IMPORT WITH SYSLIB.LOAD_FROM_HCATALOG_HDP2_1_2,
DO EXPORT WITH SYSLIB.LOAD_TO_HCATALOG_HDP2_1_2;
Example: Creating a Server Object for Kerberos
The following example creates a server object for a Kerberos-protected Hadoop cluster. It uses an authentication object named auth_cdh
that is located in the user database for the session user:
CREATE FOREIGN SERVER TD_SERVER_DB.hadoop2
EXTERNAL SECURITY INVOKER TRUSTED auth_cdh
USING
hosttype('hadoop')
server('hserver_name.example')
port('9083')
hiveport ('10000')
security('kerberos')
clustername('foo')
DO IMPORT WITH SYSLIB.LOAD_FROM_HCATALOG_CDH5_4_3,
DO EXPORT WITH SYSLIB.LOAD_TO_HCATALOG_CDH5_4_3;
34 Teradata QueryGrid: Teradata Database-to-Hadoop User Guide
