Chapter 7 Loading and Exporting Data: Application DBAs. Teradata Release 15.10
Реклама
Реклама
CHAPTER 7
Loading and Exporting Data:
Application DBAs
This chapter provides an overview of the different methods used to load data into or export data from Teradata Database. The chapter also provides guidelines on choosing the best utility for your purpose and examples of utility job performance analysis and capacity planning.
About Teradata Parallel Transporter
Teradata Parallel Transporter (PT) is an object-oriented client application that provides scalable, high-speed, and parallel data:
•
Extraction
•
Loading
•
Updating
These capabilities can be extended with customizations or third-party products.
Teradata PT uses but expands on the functionality of the traditional Teradata extract and load utilities: FastLoad, MultiLoad, FastExport, and TPump, also known as standalone utilities.
Teradata PT can load data into and export data from any accessible database object in Teradata
Database or other data store using Teradata PT operators or access modules.
About the Load Operator
The Load operator uses the Teradata FastLoad protocol to load a large volume of data at high speed into an empty table in Teradata Database.
The Load operator is typically used for the initial loading of Teradata tables. It inserts data from data streams into individual rows of a target table.
About the Update Operator
The Update operator emulates the Teradata MultiLoad utility to perform highly scalable and parallel inserts, updates, deletes, and upserts into new or pre-existing Teradata Database tables in a single pass.
The Update operator can also be used to insert new rows into the database, but it cannot perform select operations. The Update operator is not supported for column-partitioned tables.
Database Administration 135
Chapter 7: Loading and Exporting Data: Application DBAs
About Basic Teradata Query for Loading Tables
About the Export Operator
The Export operator uses the Teradata FastExport protocol to extract large amounts of data at high speed from Teradata Database using block transfers over multiple sessions. The Export operator reads data from one or more tables in Teradata Database.
About Basic Teradata Query for Loading Tables
Differences Between BTEQ and Teradata PT
BTEQ and Teradata PT can both load data into Teradata and can update tables, among other functions. Other similarities include:
•
Capacity: Both can send up to 64 KB per request and up to 1 MB per request with array support.
•
Capabilities: Both can support multiple sessions and can load, export, process DDL, and act as a command line interface.
The following table explains the major differences between BTEQ and Teradata PT:
BTEQ
Consumes less client CPU.
Loads or updates either all records or stops after x records that you specify with the REPEAT command.
Limited error handling.
Teradata PT
Consumes more client CPU.
Can also load or update only selected rows.
No checkpoint restart.
No SERIALIZE option.
Does not process multiple DML statements in a single pass of the input data.
Performs error handling and creates error log tables.
Checkpoint restart.
SERIALIZE option ensures the correct sequencing of transactions.
Can perform conditional apply or multiple apply.
Recommendation: If your job has millions of rows, use Teradata PT. If not, especially for a load job that will be used again in the future, consider both. See which performs better and causes less system overhead. BTEQ can be a good choice for smaller amounts of data, exploratory work, and DDL.
Updating Tables
Frequently, when updating the Teradata Database, you make the same change to many rows. It is helpful to eliminate the repetitive work required to update them individually. The REPEAT command serves this purpose.
136 Database Administration
Chapter 7: Loading and Exporting Data: Application DBAs
About Basic Teradata Query for Loading Tables
Updating Rows Using a Single Request
If you can describe all of the rows by some formula, then you can use a single request to update them all. The following request, for example, updates the salaries of all employees in department 600 of the database Personnel: update personnel.employee set salary=salary*1.07
where deptno=600;
Updating Rows Using Multiple Requests
If you cannot describe all of the rows by a formula, you must describe them individually. The following requests, for example, update the salaries of two employees: update personnel.employee set salary=salary*1.07
where empno = 10006; update personnel.employee set salary=salary*1.07
where empno = 10013;
This approach is convenient for a few updates, but if there were hundreds, the script would become very long. It would also be very inflexible; you would have to edit each entry to use the script again for another list of employees.
Updating Rows by Importing a File
A better approach would be to create a separate file (for example, RAISEEMP) containing two records in data format representing the employee numbers 10006 and 10013. Then use the
BTEQ IMPORT command with a REPEAT command to update the rows in the database.
Using the REPEAT Command When Importing a File
The REPEAT command appears before the request and specifies the total number of requests to be submitted.
The following BTEQ script using the BTEQ REPEAT command opens the character-format file RAISEEMP and repeats the update sequence twice:
.IMPORT data file=raiseemp
.REPEAT 2 using enumb (char(5)) update personnel.employee set salary=salary*1.07
where empno = :enumb;
For each employee, BTEQ reads a value from the RAISEEMP file for the variable called enumb and carries out the UPDATE for the row whose EmpNo equals the value of enumb.
Note: When using a file originally exported with the BTEQ EXPORT command as the source for an IMPORT command across a different platform type, ensure that the endianness type of both platforms is the same. You can verify this from the “Client Platform Byte Order” tab in the output of the SHOW CONTROLS command.
Loading Data into Teradata Database
BTEQ Input Stream
Use the following BTEQ input stream to load (import) data into the Teradata Database:
Database Administration 137
Chapter 7: Loading and Exporting Data: Application DBAs
About Reading Committed Data While Loading to the Same Table
.IMPORT DATA FILE=DeptData.imp
.REPEAT *
USING DeptNo (SMALLINT),
DeptName (VARCHAR(14)),
Loc (CHAR(3)),
MgrNo (SMALLINT)
INSERT INTO Personnel.newtable (DeptNo, DeptName,
Loc,MgrNo)
VALUES (:DeptNo, :DeptName, :Loc, :MgrNo); where:
•
The REPEAT * command repeats the next request until the data in the import file is exhausted.
•
The term newtable refers to an empty table to receive the data being loaded.
Reference Information
Information on...
BTEQ commands, including
PACK running batch jobs to submit
BTEQ commands and
Teradata SQL
Is available in...
Basic Teradata Query Reference
Basic Teradata Query Reference
Restarts and Aborts on BTEQ Jobs with Identity Column
If a restart occurs during a BTEQ import, BTEQ will resubmit the last uncompleted row insert after the system recovers from the restart. Identity column numbering will continue from there. It is possible, however, for a restart to cause duplicates because BTEQ may do a reinsert even if the previous insert has completed. Duplicates will not be detected if the target table is
MULTISET and not defined with a UPI.
If a session abort occurs during a mainframe-attached BTEQ import, the last uncompleted row insert will not be resubmitted and associated data may be lost. Associated data may also be lost if a workstation-attached BTEQ import session is aborted and there is no other session through which to resubmit the insert.
In both cases, manually restarting the import can result in duplicate rows if rows newly inserted before the session abort are not deleted unless the target table is defined with a UPI.
About Reading Committed Data While Loading to the Same Table
Users can concurrently read committed data in a table while the table is being loaded when all of these conditions are met:
138 Database Administration
Chapter 7: Loading and Exporting Data: Application DBAs
About Reading Committed Data While Loading to the Same Table
•
The table must be a load-isolated table.
You identify a table as load isolated using the ISOLATED LOADING option with CREATE
TABLE or ALTER TABLE requests. For example:
CREATE TABLE ldi_table01, WITH CONCURRENT ISOLATED LOADING FOR ALL
(col_a INT, col_b VARCHAR(32));
•
The load is performed using isolated loading.
As described later in this section, isolated loading can occur even when you don’t explicitly specify ISOLATED LOADING. If you want to ensure isolated loading use the ISOLATED
LOADING option with your SQL request (INSERT, UPDATE, MERGE, and DELETE).
For example:
INSERT WITH CONCURRENT ISOLATED LOADING INTO ldi_table01 SELECT * FROM src_Tab;
•
The SELECT request must use the LOAD COMMITTED locking option to read committed data from the table. For example:
LOCKING TABLE ldi_table01 FOR LOAD COMMITTED SELECT * FROM
ldi_table01;
You can use Teradata PT with the SQL Inserter and SQL Selector operators for load isolation because these operators perform SQL requests. (The Load and Update operators perform SQL requests when the MLOADX protocol is used; however, Teradata Database determines the protocol that is used for each job. If the Load or Update operator uses the MULTILOAD protocol, then concurrent reads will not be allowed.)
You can determine whether the ISOLATED LOADING option is being used for a load by using the DBQL step level logging facility. See DMLLoadID in
“Query Step Information Log Table:
.
If a load is performed on a load-isolated table using a SQL request without the ISOLATED
LOADING option then the default is to:
•
Allow concurrent reads on committed rows while the table is being all-AMP updated
•
Disallow concurrent reads while the table is modified with less than an all-AMP operation
Note: All SQL requests in a transaction must use the same type of concurrency as the first request in the transaction or the transaction will fail.
For details on CREATE TABLE and ALTER TABLE syntax, see SQL Data Definition Language
Syntax and Examples. For details on the SELECT, INSERT, UPDATE, MERGE, DELETE, and
LOCKING request modifier syntax, see SQL Data Manipulation Language.
For information on locking and load isolation, see SQL Request and Transaction Processing.
About Cleaning up Load Isolated Tables
Load Isolation uses row versioning to ensure that only the most recently committed data is retrieved. Logically deleted rows are created as an artifact of row versioning and, as a result, a load-isolated table typically grows faster than a table without load isolation. A DBA must physically delete the logically deleted rows at regular intervals to reclaim disk space and improve query performance.
Database Administration 139
Chapter 7: Loading and Exporting Data: Application DBAs
About FastLoad
Larger tables with more concurrent reads during updates will require more frequent cleanup.
When you collect summary statistics on your load-isolated tables, monitor the logically deleted rows (DelRowCount) as a percentage of physical rows (PhyRowCount). Perform cleanup when the table contains a significant percentage of logically deleted rows or you notice degraded query performance on the table while modifications occur.
To clean up logically deleted rows, you can use either the RELEASE DELETED ROWS option of an ALTER TABLE request or issue a stored procedure to perform cleanup on cascaded, logically deleted rows in load-isolated tables and underlying join indexes and secondary indexes.
Alter Table Cleanup Method
Use the RELEASE DELETED ROWS option in an ALTER TABLE request to interactively clean up logically deleted rows from a load-isolated table. For example:
ALTER TABLE Employee RELEASE DELETED ROWS;
For details on the syntax, see SQL Data Definition Language Syntax and Examples.
Stored Procedure Cleanup Method
Use a stored procedure to clean up logically deleted rows from multiple load-isolated tables and their underlying join indexes and secondary indexes. Use LDI_Clean_ANSI in ANSI mode and LDI_Clean in Teradata mode. The input and output is the same for both ANSI and
Teradata mode. (These stored procedures are created typically during Teradata Database installation but can be added later using the DIPDEM or DIPPOST scripts.)
The following example uses LDI_Clean in Teradata mode: call ldi_clean('database_name', 'table_name','Reset_flag_value'); where Reset_flag_value is either ‘Y’ or ‘N’ to specify whether you want to reset the incremental value of the current load to 0. For example: call ldi_clean('finance', 'acct_payable','y');
Teradata responds with the following output:
acct_payable altered, finance altered
Keep in mind the following points:
•
An exclusive lock is required while performing cleanup on a load-isolated table.
•
The DROP TABLE privilege is required to perform the cleanup.
About FastLoad
FastLoad allows you to load data from the client to your Teradata Database system. It allows fast loading of a single empty table that has no SIs, JIs, HIs, RI (FKs), or column partitioning.
FastLoad loads large amounts of data into an empty table, with or without a primary index, on Teradata Database. However, row size and the number of columns, more than any other factors, do affect performance.
140 Database Administration
Chapter 7: Loading and Exporting Data: Application DBAs
About FastExport
When you load large tables, consider the following things.
IF you are loading a large table…
with fallback without fallback
THEN…
•
Observe the suggestions for loading the large table without fallback.
•
Create the table without fallback protection.
•
Load the table.
•
Use BTEQ to alter the table to have fallback protection.
•
Initially, set session to the number of AMPs in your system. Then, experiment by reducing the number of sessions.
•
Avoid checkpointing too frequently.
•
Avoid NULLIF clauses, VAR fields, or indicator bits whenever possible.
•
Run concurrent FastLoad jobs. The maximum number of FastLoad jobs you can run is 30. The default is 5. (See Utilities for more information.)
You can also change the maximum number of jobs allowed by defining throttle rules through the Teradata Viewpoint Workload Designer portlet. For more information on FastLoads, see
Concurrent FastLoad, MultiLoad, and FastExport Jobs” on page 143
.
FastLoad does not support the following:
•
HIs
•
JIs
•
FKs
•
SIs (Both USIs and NUSIs are not supported)
•
LOBs
•
Column partitioning
For more information on this utility, see Teradata FastLoad Reference.
About FastExport
FastExport quickly exports data from a Teradata Database system to the client platform. This utility can format and export very large amounts of data very quickly. The maximum number of FastExport jobs you can run is 60. The default is 5. You can also change the maximum number of jobs allowed by defining throttle rules through the Teradata Viewpoint Workload
Designer portlet. (See “Maximum Number of Concurrent FastLoad, MultiLoad, and
FastExport Jobs” on page 143 for more information.)
To improve performance:
•
Do not use too many sessions.
•
Avoid using any option that causes the evaluation of individual fields within a layout, including NULLIF clauses and APPLY WHERE conditions.
Database Administration 141
Chapter 7: Loading and Exporting Data: Application DBAs
About MultiLoad
The following restrictions and limitations apply:
•
Exponential operators
•
Concatenated fields
•
Hexadecimal forms
For more information, see Teradata FastExport Reference.
About MultiLoad
MultiLoad allows you to upload data from the client to your Teradata Database. It operates on multiple tables simultaneously and can also insert, update, and delete data.
The maximum number of MultiLoad jobs you can run is 30. The default is 5. You can also change the maximum number of jobs allowed by defining throttle rules through the Teradata
Viewpoint Workload Designer portlet. (See “Maximum Number of Concurrent FastLoad,
MultiLoad, and FastExport Jobs” on page 143
for more information.)
To improve performance:
•
Minimize concatenation and redefinition of input data.
•
Restrict the number of NUSIs.
•
Make the PI of each data table unique or nearly unique.
•
Minimize the use of error tables.
•
Do not checkpoint too often.
•
Avoid using too many sessions.
MultiLoad does not support the following:
•
Aggregate operators
•
Exponential operators
•
Arithmetic functions
•
Concatenation of data files
•
Hexadecimal forms
•
NoPI tables
•
Foreign keys
•
Loading LOBs
•
Column-partitioned tables
For more information, see Teradata MultiLoad Reference.
You can perform the following steps as an alternative to using MultiLoad:
1
2
Create a staging table that is otherwise identical with the target table, but has no constraints, triggers, or indexes defined on it, such as a NoPI.
Import into the staging table from the external source (for example, using Fastload or
Teradata Parallel Data Pump Array INSERT operations).
142 Database Administration
Chapter 7: Loading and Exporting Data: Application DBAs
Maximum Number of Concurrent FastLoad, MultiLoad, and FastExport Jobs
3
4
5
(Optional) create an error table for the target table if error logging is desired.
Execute a MERGE into the target table using the staging table as source.
Drop the staging table and error table.
For information on NoPI tables or Teradata Parallel Data Pump Array INSERT operations, see
Database Design.
Maximum Number of Concurrent FastLoad,
MultiLoad, and FastExport Jobs
If you do not use the Teradata Viewpoint Workload Designer portlet to throttle concurrent load and unload jobs, the MaxLoadTasks and the MaxLoadAWT fields of the DBSControl determine the combined number of FastLoad, MultiLoad, and FastExport jobs that the system allows to run concurrently. The default is 5 concurrent jobs.
If you have the System Throttle (Category 2) rule enabled, even if there is no Utility Throttle defined, the maximum number of jobs is controlled by the Teradata dynamic workload management software and the value in MaxLoadTasks field is ignored.
For more information, see “MaxLoadTasks” and “MaxLoadAWT” in the chapter on DBS
Control in Utilities.
Utility Management
To manage load utilities (such as, FastLoad, MultiLoad, and FastExport) and the Teradata
ARC utility, you can classify them into a workload definition based on a combination of the following criteria:
•
Utility Type
•
“Who” criteria (such as, account name, user name, query band, and so on)
•
Database and table or view name (except for archive and restore jobs)
You can use the following utility-related Teradata Active System Management (TASM) rules to define how Teradata Database manages its workloads:
•
Utility Throttle (also called Utility Limit in the Teradata Viewpoint Workload Designer portlet). This rule enforces system wide limits on each type of the Teradata utilities, such as FastLoad, MultiLoad, FastExport, Teradata ARC, Teradata Parallel Transporter Load operator, JDBC FastLoad, and so on.
Database Administration 143
Chapter 7: Loading and Exporting Data: Application DBAs
Teradata Parallel Data Pump
•
Utility Workload Throttle. This rule enforces the limits based on the following criteria:
•
Utility Type
•
“Who” criteria (such as account name, user name, query band, and so on)
•
Database and table or view name (not available for archive or restore jobs)
To define a Utility Workload Throttle, create a workload definition with a throttle limit and the above qualification criteria (or a combination).
•
AWT Resource Limit. This rule specifies an AMP Worker Task (AWT) limit for utilities.
The rule enforces the limit based on the utility type (except the Archive and Recovery utility), request source (for example, username and account), and query band. The maximum value for the AWT Resource Limit rule is 70% of the value of the DBS Control utility field MaxAMPWorkerTasks.
•
Utility Session. This rule specifies how many utility sessions can be used for the various kinds of utilities. The supported qualification criteria are:
•
Utility Type
•
Account name, user name, query band, and so on
When the Teradata dynamic workload management software is enabled and there is no applicable user-defined Utility Session rule, the Teradata dynamic workload management software automatically uses the default session rules to select the number of sessions based on the utility type, system configuration, and optional data size.
You cannot delete default session rules. However, you can modify the utility session information (that is, the default numbers assigned to various utility kinds).
You can manage your load utilities, DSA, and the Teradata ARC utility by using Teradata
Viewpoint.
Teradata Parallel Data Pump
The Teradata Parallel Data Pump (TPump) utility allows real time updates from transactional systems into the data warehouse. TPump executes INSERT, UPDATE and DELETE requests, or a combination, to more than 60 tables at a time from the same source feed.
TPump is an alternative to MultiLoad. The benefits of TPump include:
•
Real time INSERTs and UPDATEs to more than 60 tables simultaneously
•
Low volume batch maintenance
•
Can provide continuous feed to the warehouse
The data handling functionality of TPump is enhanced by the TPump Support Environment.
In addition to coordinating activities involved in TPump tasks, it provides facilities for managing file acquisition, conditional processing, and certain Data Manipulation Language
(DML) and Data Definition Language (DDL) activities, on Teradata Database.
TPump has the following restrictions:
•
It does not support aggregate operators or concatenation of data files
144 Database Administration
Chapter 7: Loading and Exporting Data: Application DBAs
Loading Geospatial Data
•
TPump performance is severely affected by access or DBQL logging.
The TPump Support Environment enables use of variable substitution, conditional execution based on the value of return codes and variables, expression evaluation, character set selection options and more. For more information, see Teradata Parallel Data Pump Reference.
Restarts on TPump Jobs with Identity Column
TPump works on multistatement SQL requests. Each request has a specific number of statements depending on the PACK specification in the BEGIN LOAD command.
In ROBUST mode, each request is written into a restart log table.
If a restart occurs, TPump will query the restart log table and re-execute requests that are not logged. This means it may be possible for a restart to generate duplicates if an insert request is repeated. Duplicates will not be detected if the target table is not defined with a UPI.
TPump will flag an error if it is run in simple mode and the target table has an identity column
PI. This is because no restart log is used for restart recovery and duplicate rows could result if some requests are reprocessed.
For more information on this utility, see Teradata Parallel Data Pump Reference.
Loading Geospatial Data
The TDGeoImport data load program can convert geospatial data into a format that can be used by the Teradata geospatial UDTs.
TDGeoImport converts layers from ESRI, MapInfo, and TIGER/Line data sources and loads them directly into the database. This program can be used with geometries that have web representations of any size up to a maximum of 16MB.
Interpreting LOAD Utility Status
When observing the progress of load, archive, export, and other jobs, remember the following:
•
The system uses LSN to calculate the impact of all sessions.
•
All work the system performs during a session is charged to that session.
Because not all sessions are used during an operation, some sessions with the same LSN and user name may not report any activity.
The Query Session utility (QRYSESSN) can provide detailed status of each session involved in the load task.
Database Administration 145
Chapter 7: Loading and Exporting Data: Application DBAs
Choosing the Best Utility for Your Purpose
Choosing the Best Utility for Your Purpose
Some of the client utilities have similar functions. However, to optimize performance on resource utilization, it is important to choose the best utility for your job. Note that the following section does not include Teradata Parallel Transporter.
Definitions
To better understand the guidelines in this section, refer to the definitions in the following table.
Term
Small table
Large table
Small number of rows
Moderate number of rows
Large number of rows
Meaning
<1 million rows
>100 million rows
<5% of total rows
20 - 30% of total rows
40 - 50% of total rows
Guidelines for Inserting Rows
The following table provides some guidelines you can use to make the best choice of utilities when inserting rows. Keep in mind that you may find a better way through experimenting.
Task
Insert a small number of rows into an empty table
Insert a large number of rows into an empty table
Insert a small number of rows into a small populated table
First Choice
FastLoad
FastLoad
BTEQ, MultiLoad, TPump or
MERGE when the transaction density is too low for satisfactory performance
TPump, BTEQ, or MERGE Insert a small number of rows into a large populated table
Insert a moderate number of rows into a large populated table
Insert a large number of rows into a large populated table
MultiLoad or MERGE
MultiLoad or MERGE
Second Choice
TPump/BTEQ
TPump
TPump when the transaction density is too low for satisfactory performance
1
2
FastLoad rows into an empty table.
INSERT...SELECT rows into an empty new table.
146 Database Administration
Chapter 7: Loading and Exporting Data: Application DBAs
Choosing the Best Utility for Your Purpose
Task
Insert a large number of rows into multiple populated tables
Insert a large number of rows when logged on session are near their full capacity
Insert a large number of rows when the specific order of updates is important
Insert a large number of rows into multiple empty tables
First Choice
MultiLoad
INSERT…SELECT or MERGE
Tpump
Multiple FastLoads
Second Choice
TPump
Tpump
INSERT…SELECT or MERGE
MultiLoad
Guidelines for Deleting Rows
The following table provides some guidelines you can use to make the best choice of utilities when deleting rows. Keep in mind that you may find a better way through experimenting.
Task
Delete a small number of rows from any table
Delete a large number of rows from a large table
Perform DELETE operations on a large number of rows on multiple tables
Delete all rows from a table
First Choice
TPump
MultiLoad
MultiLoad
BTEQ
In this case, Teradata marks the table header with an indicator to say that the table is empty. Assuming there are no locks on the table when this delete is requested, the delete will complete in a second or two.
Second Choice
BTEQ
BTEQ
TPump
Database Administration 147
Chapter 7: Loading and Exporting Data: Application DBAs
Choosing the Best Utility for Your Purpose
Task
Delete some rows from a table
First Choice
BTEQ
Space permitting, it can also be an option to create an empty table, insert into this table the rows that you wish to retain and drop the original table and then rename the new table to the name of the original table.
Second Choice
MultiLoad deletes are good for deleting large volumes of data, but not all data, from a table.
MultiLoad reads the data block by block. It will “touch” each block consumed by the table and remove rows as appropriate, and replace the block.
This can be very efficient and will beat an
SQL delete in many cases as there is no rollback logging. However, MultiLoad must complete or else the table will remain in an unstable state. For MultiLoad to work best, drop SIs, run the MultiLoad delete, and then replace the indexes. This is quicker and ensures that the work table is small.
Note: You cannot delete rows from a table using the SQL MERGE statement.
Guidelines for Other Batch Operations
When updating, exporting, or doing other batch operations, use the guidelines in the following table to consider the best choice for your job.
Task
Merging a large number of rows when logged on session are near their full capacity
Merging a large number of rows when the specific order of updates is important
Export a small number of rows from any table
Export a moderate or large number of rows from a large table
Update a small number of rows in any table
Update a large number of rows in a large table
Perform multiple DML
(INSERT, UPDATE, DELETE) operations on a small number of rows on multiple tables
First Choice Second Choice
INSERT…SELECT or MERGE Tpump
Tpump
BTEQ
FastExport
TPump
MultiLoad
MultiLoad or TPump when the transaction density is too low for satisfactory performance
INSERT…SELECT or MERGE
FastExport
BTEQ
BTEQ
BTEQ insert table into a new one.
BTEQ
148 Database Administration
Chapter 7: Loading and Exporting Data: Application DBAs
Utility Job Performance Analysis and Capacity Planning
Task
Perform multiple DML
(INSERT, UPDATE, DELETE) operations on a large number of rows on multiple tables
First Choice
MultiLoad
Copy one or several tables to a different Teradata Database
Copy all tables in the Teradata
Database system to a different
Teradata Database system
ARC
ARC
Load an identity column table TPump (in ROBUST mode)
Load the data into staging table
Teradata PT Load Operator
Second Choice
TPump
FastExport and FastLoad
BTEQ .IMPORT
Teradata PT Update Operator
Utility Job Performance Analysis and Capacity
Planning
You can query the DBQL utility job log table, DBC.DBQLUtilityTbl, to analyze the performance of load or export utility jobs and do capacity planning. This table is populated if you use the WITH UTILITYINFO option with a BEGIN/REPLACE QUERY LOGGING request. For details on the contents of DBQLUtilityTbl and the corresponding view
QryLogUtility[V], see “Utility Job Log Table: DBQLUtilityTbl” on page 337 and
Utility Job Log Table: QryLogUtility[V]” on page 343
.
Examples of Using the DBQL Utility Job Log Table
You can use DBQLUtilityLogTbl to diagnose utility performance issues and do capacity planning. The following are just some examples of how to use the table.
Investigating Why a Job Ran Slower Today
In this scenario, a FastLoad job submitted today took significantly longer than yesterday. You want to determine the root cause. Although the example is specific to Fast-
Load, you can use this general approach for other utilities.
1
Retrieve the rows for the jobs yesterday and today from DBQLUtilityTbl using the following criteria:
•
Job attributes that uniquely identify this FastLoad job. This criteria returns all instances of this FastLoad job. Possible job attributes include:
•
UtilityName = 'FASTLOAD'
•
Username
•
Query band
•
Utility request
Database Administration 149
Chapter 7: Loading and Exporting Data: Application DBAs
Utility Job Performance Analysis and Capacity Planning
2
3
•
Job end time: either today or yesterday.
Compare various column values between the two rows to determine the possible causes.
Suggestions include:
a
Compare RowsInserted values. If the job today loaded significantly more rows, this could be the cause. If not, proceed to the next step.
b
Compare DelayTime values. An increase indicates the job today was delayed longer by
TASM.
c
Compare the elapsed time of each phase (PhasexEndTime - PhasexStartTime) to identify the phase with a significant increase in elapsed time.
d
If the phase with a significant increase in elapsed time is the Acquisition phase, check for an increase in MaxDataWaitTime, which might be caused by more load on the client machine or slower network response time.
e
Compare the resource usages of the phase with a significant increase in elapsed time to find additional clues. For example:
•
An increase in PhasexMaxCPUTime may indicate data skew.
•
An increase in Phase1 messages (Phase1BlockCount) may indicate that a smaller message size was used.
Examine related query log data in DBQLogTbl. To select the desired data, use:
•
LSN to select all rows of a particular job.
•
PhasexStartTime and PhasexEndTime to select rows of a specific phase.
Performance Anomaly Detection
In this scenario, you want to detect job instances whose elapsed time is 50% longer than the last 30-day average for the same job. This approach can be used to detect anomalies for other metrics, such as CPU time or I/Os.
1
2
Select rows from DBQLUtilityTbl for jobs ending within the last 30 days and store them in a temporary table called Last30dayJobInstances.
CREATE VOLATILE TABLE Last30dayJobInstances AS
(SELECT U.*,
SUBSTRING(UtilityRequest FROM 1 FOR 100) AS UtilReqShort
FROM DBC.DBQLUtilityTbl U
WHERE CAST(JobEndTime AS date) >= DATE-30)
WITH DATA
PRIMARY INDEX (UtilityName, UserName)
ON COMMIT PRESERVE ROWS;
Use the Last30dayJobInstances table to calculate the last 30-day average elapsed time of each job, grouped by attributes such as utility name, user name, and utility request. Store the results in another temporary table called Last30dayAverage. Each row contains the average elapsed time and job attributes.
CREATE VOLATILE TABLE Last30dayAverage AS
(SELECT AVERAGE((JobEndTime - JobStartTime) HOUR TO SECOND)
AS AvgJobElapsedTime,
UtilityName, UserName,
SUBSTRING(UtilityRequest FROM 1 FOR 100) AS UtilReqShort
150 Database Administration
Chapter 7: Loading and Exporting Data: Application DBAs
Utility Job Performance Analysis and Capacity Planning
3
FROM Last30dayJobInstances J
GROUP BY UtilityName, UserName,
SUBSTRING(UtilityRequest FROM 1 FOR 100))
WITH DATA
PRIMARY INDEX (UtilityName, UserName)
ON COMMIT PRESERVE ROWS;
Select job instances whose elapsed time is more than 50% of the average using the
Last30dayJobInstances and Last30dayAverage tables.
SELECT I.*
FROM Last30dayJobInstances I,
Last30dayAverage A
WHERE I.UtilityName = A.UtilityName
AND I.UserName = A.UserName
AND I.UtilReqShort = A.UtilReqShort
AND (
(EXTRACT(HOUR FROM ((I.JobEndTime - I.JobStartTime) HOUR TO SECOND)) * 3600) +
(EXTRACT(MINUTE FROM ((I.JobEndTime - I.JobStartTime) HOUR TO SECOND)) * 60) +
(EXTRACT(SECOND FROM ((I.JobEndTime - I.JobStartTime) HOUR TO SECOND)))
) > (
((EXTRACT(HOUR FROM A.AvgJobElapsedTime) * 3600) +
(EXTRACT(MINUTE FROM A.AvgJobElapsedTime) * 60) +
(EXTRACT(SECOND FROM A.AvgJobElapsedTime))
) * 1.5);
Capacity Planning
In this scenario, you calculate these monthly metrics for each utility for the previous
12 months:
•
Number of job instances
•
Average job elapsed time
•
Total processing volume (byte counts and row counts)
•
Average processing volume
•
Average throughput (rows per second and bytes per second)
SELECT TRUNC(CAST(JobEndTime AS DATE), 'RM') JobMonth, UtilityName,
COUNT(*) AS NumJobs,
AVG(
(EXTRACT(HOUR FROM ((JobEndTime - JobStartTime) HOUR TO SECOND)) * 3600) +
(EXTRACT(MINUTE FROM ((JobEndTime - JobStartTime) HOUR TO SECOND)) * 60) +
(EXTRACT(SECOND FROM ((JobEndTime - JobStartTime) HOUR TO SECOND)))
) AS AvgJobTime,
SUM(RowsInserted), AVG(RowsInserted),
SUM(RowsUpdated ), AVG(RowsUpdated ),
SUM(RowsDeleted ), AVG(RowsDeleted ),
SUM(RowsExported), AVG(RowsExported),
SUM(Phase1ByteCount), AVG(Phase1ByteCount),
SUM(Phase2ByteCount), AVG(Phase2ByteCount),
AVG(RowsInserted) / AvgJobTime AS InsRowsPerSec,
AVG(RowsUpdated ) / AvgJobTime AS UpdRowsPerSec,
AVG(RowsDeleted ) / AvgJobTime AS DelRowsPerSec,
AVG(RowsExported) / AvgJobTime AS ExpRowsPerSec,
AVG(Phase1ByteCount) / AvgJobTime AS AvgPhase1BytesPerSec,
AVG(Phase2ByteCount) / AvgJobTime AS AvgPhase2BytesPerSec
FROM (SELECT *
FROM DBC.DBQLUtilityTbl U
Database Administration 151
Chapter 7: Loading and Exporting Data: Application DBAs
Utility Job Performance Analysis and Capacity Planning
WHERE CAST(JobEndTime AS DATE) >= ADD_MONTHS(TRUNC(CURRENT_DATE, 'RM'), -12)
AND CAST(JobEndTime AS DATE) < TRUNC(CURRENT_DATE, 'RM') ) AS OneYearLog
GROUP BY JobMonth, UtilityName
ORDER BY JobMonth, UtilityName;
152 Database Administration
Изтегляне
Публичната връзка е актуализирана
Публичната връзка към вашия чат е актуализирана. You may view and manage your chat links after registration here.
Публична връзка към документа
Реклама
