Introduction. Teradata QueryGrid

PDF

Documento

CHAPTER 2

Syntax for Teradata QueryGrid: Teradata

Database-to-Hadoop

Introduction

This chapter describes the syntax and options for the Teradata-to-Hadoop connector, and provides examples of their use. It includes DDL statements, and information about DML and DCL statements.

If you are a DBA, or a DBA has already granted you the privileges needed to create foreign servers, we recommend that you start by reading

CREATE FOREIGN SERVER .

ALTER FOREIGN SERVER

Purpose

Modifies the parameters of an existing server object.

Syntax

Teradata QueryGrid: Teradata Database-to-Hadoop User Guide 11

Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop

ALTER FOREIGN SERVER

Syntax Elements

server_name

The name given to the foreign server object.

EXTERNAL SECURITY

Associates an authorization object with the foreign server. The authorization stores the encrypted credentials for a user as a database object. The Teradata QueryGrid connector passes the credentials in the authorization to the remote platform identified by the foreign server when the foreign server is accessed.

You must use EXTERNAL SECURITY TRUSTED for Teradata QueryGrid: Teradata

Database-to-Hadoop when the Hadoop platform is protected by an external security system, such as Kerberos, for example.

INVOKER

DEFINER

INVOKER is a keyword that indicates that the associated authorization must be present in the user database at the time that the foreign server is accessed.

Note: The user database is the database that was created for the user in the Teradata system when the user account was created.

DEFINER is a keyword that indicates that the associated authorization must be present in the database that contains the foreign server when the foreign server is accessed.

Note: The DEFAULT keyword that can be used with DEFINER in CREATE

AUTHORIZATION and REPLACE AUTHORIZATION statements is not needed in association with a foreign server.

You must use either INVOKER TRUSTED or DEFINER TRUSTED if the remote platform uses an external security system (such as Kerberos, for example) for authentication.

12 Teradata QueryGrid: Teradata Database-to-Hadoop User Guide

Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop

ALTER FOREIGN SERVER

TRUSTED

A keyword that indicates the associated authorization object was created as TRUSTED.

authorization_name

Specifies the name of the authorization object to be used when the foreign server is accessed.

Modify options

ADD

Use to:

• add or replace a global name value pair that is used to define the server object

• add an IMPORT or EXPORT table operator. If you want to replace a table operator that is already associated with the foreign server you must first drop the table operator before adding the new one.

• add or replace a local name value pair that is used with an IMPORT or EXPORT table operator

name('value')

The name value pair or pairs that you want to add or modify.

Note that in the description of the name value pairs:

• The label "Server only" indicates that the name value pair must follow the syntax

ADD name('value')

• The label "Import only" indicates that the name value pair must be specified after the

IMPORT keyword.

• The label "Export only" indicates that the name value pair must be specified after the

EXPORT keyword.

• Unlabeled name value pairs may be specified in any part of the ADD syntax. If specified as

ADD name('value')

, the name value pair will be applied to the server as a whole.

For descriptions of the name value pairs used with the server object, see Required

Name Value Pairs

and

Optional Name Value Pairs

IMPORT

Indicates that you are going to act on the operator that is used to import data into

Teradata Database.

EXPORT

Indicates that you are going to act on the operator that is used to export data out of

Teradata Database.

operator_name

The name of the table operator that you want to use.

For more information about the table operators used with the server object, see

CREATE FOREIGN SERVER

Drop options

DROP

• to drop a global name value pair that was used to define a server object. You need only specify the name to drop the pair.

Teradata QueryGrid: Teradata Database-to-Hadoop User Guide 13

Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop

ALTER FOREIGN SERVER

• to drop an IMPORT or EXPORT table operator that was associated with a server definition. When you drop a table operator all related name value pairs are also dropped.

• to drop a local name value pair that was used with an IMPORT or EXPORT table operator. You need only specify the name to drop the pair.

name

When used alone, name is the name of the name value pair that you want to drop.

For more information about the name value pairs used with the server object, see

Required Name Value Pairs

and

Optional Name Value Pairs

IMPORT

Indicates that you are going to act on the operator that is used to import data into

Teradata Database.

EXPORT

Indicates that you are going to act on the operator that is used to export data out of

Teradata Database

Required Name Value Pairs

These name value pairs are required to create a functioning foreign server object. Additional optional name value pairs may be required to create a foreign server for a specific implementation.

hosttype

Server only.

port

For Teradata QueryGrid: Teradata Database-to-Hadoop, this is

('hadoop')

Server only. The server port number for the Hive Metastore; typically this is 9083.

server

Server only. The DNS host name or IP address for the Apache Hive Metastore

(hive.metastore.uris). You can use an application, such as Ambari, to obtain this value.

Optional Name Value Pairs

These name value pairs are optional. However, a particular server implementation may require you to define some of these name value pairs. For example, a foreign server must be defined with a Hive port to support queries that access Hive Server2.

clustername

Required when using security('kerberos')

. Specifies the directory name that stores the JAR file that contains the configuration files (core-site.xml, hdfs-site.xml, hive-site.xml, hive-site.xml, and yarn-site.xml) for the Hadoop cluster to be accessed.

This directory was set up during the Hadoop client installation on the Teradata nodes.

For example, you would use the name value pair clustername('yourcluster')

if the files are stored as follows:

• yourcluster/

• yourcluster/core-site.xml

• yourcluster/hdfs-site.xml

• yourcluster/hive-site.xml

• yourcluster/yarn-site.xml

14 Teradata QueryGrid: Teradata Database-to-Hadoop User Guide

Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop

ALTER FOREIGN SERVER

compression_codec

Export only. Specifies the type of compression to use for the exported data. The default is no compression.

The supported compression types are based on the compression codecs configured on the Hadoop system.

Note: Snappy is supported but you must specify it in hadoop_properties using

<orc.compression=SNAPPY> as the argument. Snappy is supported only for the

ORCFile file type.

You must specify the full name for the compression codec as follows:

• org.apache.hadoop.io.compress.DefaultCodec

• org.apache.hadoop.io.compress.GzipCodec

• org.apache.hadoop.io.compress.BZip2Codec

dbname

The name of the user’s database.

This parameter is optional. You can specify a dbname value in the foreign server to limit its scope to a specific database. The value specified in the USING clause in the CREATE

FOREIGN SERVER syntax overrides any corresponding value specified directly in the user query.

default_string_size

Size at which data imported from or exported to Hadoop String columns is truncated.

When applied to the import operator, the value represents the maximum number of

Unicode characters to import, and defaults to 2048 characters. When applied to the export operator, the value represents the maximum number of bytes to export, and defaults to 4096 bytes. Teradata QueryGrid silently truncates the String columns at the default value set in default_string_size.

hadoop_properties ('<property1=value>, <property3=value1,value2>')

Sets specific properties used to interact with Hadoop. If there are multiple arguments, you must delimit them with angle brackets. If there is only one argument, you can omit the angle brackets. For example, the syntax for the hadoop_properties clause for a High

Availability (HA) target supports an updated syntax where multiple values can be included. In this case the properties must be enclosed by left and right angle brackets.

No spaces are allowed within or between arguments.

The High Availability hadoop properties are defined based on the name service that is defined on your hadoop server. For example, if you have the following Hadoop properties: hadoop_properties('

<dfs.client.use.datanode.hostname=true>

,<dfs.datanode.usedatanode.hostname=true>

,<dfs.nameservices=MYCOMPANY_HADOOP02>

,<dfs.ha.namenodes.MYCOMPANY_HADOOP02=nn1,nn2>

,<dfs.namenode.rpc-address.MYCOMPANY_HADOOP02.nn1=hdp230-2:8020>

,<dfs.namenode.rpc-address.MYCOMPANY_HADOOP02.nn2=hdp230-3:8020>

,<dfs.client.failover.proxy.provider.MYCOMPANY_HADOOP02=org.apache.

hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider>')

In this example, you would make the following replacements:

• Replace

MYCOMPANY_HADOOP02

with your own name service ID.

Teradata QueryGrid: Teradata Database-to-Hadoop User Guide 15

Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop

ALTER FOREIGN SERVER

• Replace hdp230-2

and hdp230-3

with your own namenode hostnames.

It may also be necessary for you to replace nn1

and nn2

in the example above with your own namenode aliases. To verify, check the following property in your hdfssite.xml

file:

<name>dfs.ha.namenodes.MYCOMPANY_HADOOP02</name>

<value>namenode10,namenode66</value>

</property>

In this case, you would make the following replacements:

• Replace nn1

with namenode10

• Replace nn2

with namenode66

In most cases, you should set the dfs.client.use.datanode.hostname property to true. If you have a setup where your TPA nodes are on one BYNET, the Hadoop cluster is on another BYNET, and they are communicating with one another via Ethernet, then you should also set the dfs.datanode.usedatanode.hostname property to true.

hiveserver

The DNS host name or IP address of Hive Server2. This is used when a query results in the use of the HCTAS and HDROP procedures or FOREIGN TABLE SELECT in Hive.

(You can use an application, such as Ambari, to obtain this value.) If no value is specified for hiveserver then the value for server is used.

hiveport

The port for access to the Hive Server2; typically this is 10000. You can use an application, such as Ambari, to obtain this value.

merge_hdfs_files

Export only. Indicates that files under the same partition should be merged whenever possible. The default is to not merge. A value of TRUE means that files will be merged.

row_count_report_freq

The frequency with which byte count is updated in DBQL. The default is every 100 rows. You can adjust this to a larger value if the update frequency is too resource intensive.

security

Specifies the name of the external security system used for authentication on the

Hadoop cluster. This parameter is required when an external security system is in use.

The default is no security. Valid values are:

• kerberos

• ldap

Note: Teradata QueryGrid: Teradata Database-to-Hadoop supports only Kerberos authentication when used with Cloudera CDH 5.4; the use of LDAP on Cloudera CDH

5.4 is not supported.

tablename

The name of the table to be imported or exported.

This parameter is optional. You can specify a tablename value in the foreign server to limit its scope to a specific table. The value specified in the USING clause in the

CREATE FOREIGN SERVER syntax overrides any corresponding value specified directly in the user query.

16 Teradata QueryGrid: Teradata Database-to-Hadoop User Guide

Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop

ALTER FOREIGN SERVER

temp_dbname

Import only. The value is the name of the Hadoop database to use to store temporary

Hive staging tables.

This parameter is optional. You should consider using temp_dbname when planning to use FOREIGN TABLE syntax on a foreign server or when a foreign server is set up to use usenativequalification. If no database is specified with temp_dbname, then the default Hadoop database is used.

To use the specified database, it must exist when the foreign server is created or altered to use the database. The session user must have create and write permission to the database. If multiple users use the same foreign server then the Hadoop administrator may want to consider setting up Hive authorization in such a way that temporary tables cannot be read by another user.

transformformatting

Import only. When set to 'true' it indicates that an array list data is formatted appropriately, so that it can be cast directly into a Teradata array column type based on the appropriate data type.

This parameter is optional. The value specified in the USING clause in the CREATE

FOREIGN SERVER syntax overrides any corresponding value specified directly in the user query.

updatestatistics

Export only. Indicates that the LOAD_TO_HCATALOG_abcn_n_n operator updates the table statistics after all the data has been loaded into the target Hive table. Valid values are 'true' and 'false'. A value of true means that the table statistics are updated.

Note: You must also have set hive.stats.autogather to true in your hive-site.xml file for updatestatistics to work properly.

usenativequalification

Import only. A value of 'true' indicates that SELECT queries should be pushed down to

Hive as much as possible. When a foreign server uses usenativequalification, Teradata

Database examines the following conditions:

• Hive table data size is large and there are qualifying predicates on non-partitioned

Hive columns. Large is defined as having a number of splits that is larger than the number of Teradata Database nodes.

• The queried Hive object is a view.

When either of the two conditions are met, Teradata Database constructs a Hive query from the Hive object name, the referenced columns, and the qualifying predicates. It then creates a Hive staging table (in the database specified by temp_dbname) from the constructed query and retrieves data in the staging table. The staging table is dropped after all data has been retrieved.

Valid values are 'true' and 'false.'

For queries that involve joins between two HCatalog tables, Teradata Database brings the data into Teradata spool and joins them in the database. The join is not pushed into

Hive. For example, the join syntax in the following query requires the manual

FOREIGN TABLE SELECT to be accomplished in Hive:

SELECT h1.c1, h2.c2 FROM h1@hadoop1, h2@hadoop1 WHERE h1.id = h2.id ;

Teradata QueryGrid: Teradata Database-to-Hadoop User Guide 17

Chapter 2 Syntax for Teradata QueryGrid: Teradata Database-to-Hadoop

ALTER FOREIGN SERVER

username

The name of the Hadoop user's credential. This option is ignored when the security name value pair is defined for the foreign server. If no username value and no security value are defined, the foreign server uses the name of user making the request. (This is the Teradata Database user name in capital letters.)

No password is associated with a Hadoop user. HDFS and Hive check for a user name for access. If no user name is specified, then foreign server supplies the name of the session user. If HDFS and Hive are not configured for file permissions, then the user name is optional.

Required Privileges

You must have DROP SERVER privilege on the TD_SERVER_DB database or on the specified foreign server to modify the foreign server object. If you are modifying the table operators that are associated with the server, or adding a table operator, you must also have

EXECUTE FUNCTION and SELECT privileges on the specified table operators.

Examples of Using ALTER FOREIGN SERVER

Example: Adding a New Attribute

The following example adds a new attribute to an existing server object. In this example,

INSERT and SELECT operations are limited to the table named cardata.

ALTER FOREIGN SERVER hadoop2

ADD tablename('cardata') ;

Example: Defining an EXPORT Option for an Existing Server

The following example defines an EXPORT table operator to an existing foreign server object :

ALTER FOREIGN SERVER hive_metastore_server

ADD EXPORT WITH SYSLIB.LOAD_TO_HCATALOG_HDP2_3_0 USING

merge_hdfs_files('true')