Working with AWS DMS Tasks. amazon web services AWS Database Migration Service
Publicitate
Publicitate
AWS Database Migration Service User Guide
Working with AWS DMS Tasks
An AWS Database Migration Service (AWS DMS) task is where all the work happens. You use tasks to specify what tables and schemas to use for your migration and to apply specific replication requirements to a database migration. To create a replication task, you must have at least one source and one target database to work with AWS DMS. You must also have a replication instance set up. Then, you can configure the task using task settings and specify tables to be replicated.
Once you create a task, you can run it immediately. The target tables with the necessary metadata definitions are automatically created and loaded, and you can specify that the CDC replication process be started. You can monitor, stop, or restart replication tasks using the AWS DMS console, AWS CLI, or AWS
DMS API.
The following are features you can use when performing an AWS DMS task
Task
Choosing a Task Migration Method
When you create a task, you specify the migration method, which includes either a full load of the existing data, a full load of the data plus ongoing changes, or just a replication of the ongoing changes.
Running Multiple Tasks
In some migration scenarios, you might have to create several migration tasks.
Modifying a Task
When a task is stopped, you can modify the settings for the task.
Reloading Tables During a Task
You can reload a table during a task if an error occurs during the task.
Using Table Mapping
Relevant Documentation
Migration Methods for AWS Database Migration
Creating Multiple Tasks (p. 121)
Reloading Tables During a Task (p. 120)
Selection Rules
API Version API Version 2016-01-01
118
AWS Database Migration Service User Guide
Modifying a Task
Task
Table mapping uses several types of rules to specify the data source, source schema, data, and any transformations that should occur during the task.
Relevant Documentation
Selection Rules and Actions (p. 138)
Transformation Rules
Transformation Rules and Actions (p. 139)
Applying Filters
You can use source filters to limit the number and type of records transferred from your source to your target.
For example, you can specify that only employees with a location of headquarters are moved to the target database. You apply filters on a column of data.
Applying Task Settings
Each task has settings that you can configure according to the needs of your database migration. You create these settings in a JSON file or, with some settings, you can specify the settings using the AWS DMS console.
Monitoring a Task
Task Settings for AWS Database Migration Service
There are several ways to get information on the performance of a task and the tables used by the task.
Monitoring AWS Database Migration Service Tasks (p. 148)
For more information, see the following topics:
Topics
•
•
Reloading Tables During a Task (p. 120)
•
Creating Multiple Tasks (p. 121)
•
Migration Methods for AWS Database Migration Service (p. 122)
•
Task Settings for AWS Database Migration Service Tasks (p. 122)
•
Using Table Mapping with an Task to Select and Filter Data (p. 133)
Modifying a Task
You can modify a task if you need to change the task settings, table mapping, or other settings. You modify a task in the DMS console by selecting the task and choosing
Modify. You can also use the AWS
CLI or AWS DMS API command
ModifyReplicationTask
.
There are a few limitations to modifying a task. These include:
• You cannot modify the source or target endpoint of a task.
• You cannot change the migration type from CDC to either Full_Load or Full_Load_and_CDC.
• You cannot change the migration type from Full Load to either CDC or Full_Load_and_CDC.
• A task that have been run must have a status of Stopped or Failed to be modified.
API Version API Version 2016-01-01
119
AWS Database Migration Service User Guide
Reloading Tables During a Task
Reloading Tables During a Task
While a task is running, you can reload a target database table using data from the source. You may want to reload a table if, during the task, an error occurs or data changes due to partition operations (for example, when using Oracle). You can reload up to 10 tables from a task.
To reload a table, the following conditions must apply:
• The task must be running.
• The migration method for the task must be either Full Load or Full Load with CDC.
• Duplicate tables are not allowed.
AWS Management Console
To reload a table using the AWS DMS console
1. Sign in to the AWS Management Console and choose AWS DMS. Note that if you are signed in as an AWS Identity and Access Management (IAM) user, you must have the appropriate permissions to access AWS DMS. For more information on the permissions required, see
.
2. Select Tasks from the navigation pane.
3. Select the running task that has the table you want to reload.
4. Click the Table Statistics tab.
API Version API Version 2016-01-01
120
AWS Database Migration Service User Guide
Creating Multiple Tasks
5. Select the table you want to reload. Note that if the task is no longer running, you will not be able to reload the table.
6. Choose Drop and reload table data.
When AWS DMS is preparing to reload a table, the console changes the table status to
The table is being
reloaded.
Creating Multiple Tasks
In some migration scenarios, you might have to create several migration tasks. Note that tasks work independently and can run concurrently; each task has its own initial load, CDC, and log reading process.
Tables that are related through data manipulation language (DML) must be part of the same task.
Some reasons to create multiple tasks for a migration include the following:
• The target tables for the tasks reside on different databases, such as when you are fanning out or breaking a system into multiple systems.
• You want to break the migration of a large table into multiple tasks by using filtering.
Note
Because each task has its own change capture and log reading process, changes are not coordinated across tasks. Therefore, when using multiple tasks to perform a migration, make sure that source transactions are wholly contained within a single task.
API Version API Version 2016-01-01
121
AWS Database Migration Service User Guide
Migration Methods
Migration Methods for AWS Database Migration
Service
AWS Database Migration Service can migrate your data in several ways:
• Migrating Data to the Target Database – This process creates files or tables in the target database, automatically defines the metadata that is required at the target, and populates the tables with data from the source. The data from the tables is loaded in parallel for improved efficiency. This process is the Migrate existing data option in the AWS console and is called
Full Load
in the API.
• Capturing Changes During Migration – This process captures changes to the source database that occur while the data is being migrated from the source to the target. When the migration of the originally requested data has completed, the change data capture (CDC) process then applies the captured changes to the target database. Changes are captured and applied as units of single committed transactions, and several different target tables can be updated as a single source commit.
This approach guarantees transactional integrity in the target database. This process is the
Migrate
existing data and replicate ongoing changes option in the AWS console and is called full-load-andcdc
in the API.
• Replicating Only Data Changes on the Source Database – This process reads the recovery log file of the source database management system (DBMS) and groups together the entries for each transaction. If AWS DMS can't apply changes to the target within a reasonable time (for example, if the target is not accessible), AWS DMS buffers the changes on the replication server for as long as necessary. It doesn't reread the source DBMS logs, which can take a large amount of time. This process is the Replicate data changes only option in the AWS DMS console.
Note
If you restart a task, any tables that have not completed their initial load are restarted.
Task Settings for AWS Database Migration Service
Tasks
Each task has settings that you can configure according to the needs of your database migration. You create these settings in a JSON file or, with some settings, you can specify the settings using the AWS
DMS console.
There are several main types of task settings:
•
Target Metadata Task Settings (p. 123)
•
Full Load Task Settings (p. 123)
•
Logging Task Settings (p. 124)
•
Control Table Task Settings (p. 125)
•
Change Processing Tuning Settings (p. 128)
•
Stream Buffer Task Settings (p. 128)
•
Change Processing DDL Handling Policy Task Settings (p. 129)
•
Error Handling Task Settings (p. 129)
To see an example JSON file with sample task settings, see
.
API Version API Version 2016-01-01
122
AWS Database Migration Service User Guide
Target Metadata Task Settings
Target Metadata Task Settings
Target metadata settings include the following:
•
TargetSchema
– The target table schema name. If this metadata option is empty, the schema from the source table is used. AWS DMS automatically adds the owner prefix for the target database to all tables if no source schema is defined. This option should be left empty for MySQL-type target endpoints.
•
LOB settings
– Settings that determine how large objects (LOBs) are managed. If you set
SupportLobs=true
, you must set one of the following to true
:
•
FullLobMode
– If you set this option to true
, then you must enter a value for the
LobChunkSize option. Enter the size, in kilobytes, of the LOB chunks to use when replicating the data to the target.
The
FullLobMode
option works best for very large LOB sizes but tends to cause slower loading.
•
LimitedSizeLobMode
– If you set this option to true
, then you must enter a value for the
LobMaxSize option. Enter the maximum size, in kilobytes, for an individual LOB.
•
LoadMaxFileSize
– An option for PostgreSQL and MySQL target endpoints that defines the maximum size on disk of stored, unloaded data, such as .csv files. This option overrides the connection attribute.
You can provide values from 0, which indicates that this option doesn't override the connection attribute, to 100,000 KB.
•
BatchApplyEnabled
– Determines if each transaction is applied individually or if changes are committed in batches. The default value is false
.
If set to true
, AWS DMS commits changes in batches by a pre-processing action that groups the transactions into batches in the most efficient way. Setting this value to true
can affect transactional integrity, so you must select
BatchApplyPreserveTransaction
in the
ChangeProcessingTuning
section to specify how the system handles referential integrity issues.
If set to false
, AWS DMS applies each transaction individually, in the order it is committed. In this case, strict referential integrity is ensured for all tables.
When LOB columns are included in the replication,
BatchApplyEnabled can only be used in
Limited-
size LOB mode.
Full Load Task Settings
Full load settings include the following:
• To enable or disable full load and change data capture (CDC), set the following:
•
FullLoadEnabled
– Set this option to true
if the task needs to perform a full load on target tables.
•
ApplyChangesEnabled
– Set this option to true
if the task needs to keep target tables up-to-date by applying changes using CDC.
If both
FullLoadEnabled=true
and
ApplyChangesEnabled=true
, the task loads the tables first and then keeps applying incoming changes. While updating a task, you can use these settings only to toggle on and off full load or CDC, or both.
• To indicate how to handle loading the target at full-load startup, specify one of the following values for the
TargetTablePrepMode
option:
•
DO_NOTHING
– Data and metadata of the existing target table are not affected.
•
DROP_AND_CREATE
– The existing table is dropped and a new table is created in its place.
•
TRUNCATE_BEFORE_LOAD
– Data is truncated without affecting the table metadata.
• To delay primary key or unique index creation until after full load completes, set the
CreatePkAfterFullLoad
option.
When this option is selected, you cannot resume incomplete full load tasks.
API Version API Version 2016-01-01
123
AWS Database Migration Service User Guide
Logging Task Settings
• For full load and CDC-enabled tasks, you can set the following
Stop task after full load completes options:
•
StopTaskCachedChangesApplied
– Set this option to true
to stop a task after a full load completes and cached changes are applied.
•
StopTaskCachedChangesNotApplied
– Set this option to true
to stop a task before cached changes are applied.
• To allow tables to resume processing from the point of interruption during a full load, set the
ResumeEnabled
option to true
. Otherwise, table loading restarts from the beginning.
Allowing table loading to restart from the point of interruption can cause a slowdown in full-load processing. However, starting extremely large tables from the beginning can cause long delays in completing the full load. In addition, for a table load to be restarted from the point of interruption, the table must have a unique index, and we also recommend that it have a clustered primary key.
If
ResumeEnabled
is set to true
, the following options should be set:
•
ResumeMinTableSize
– Indicates which tables should resume loading when the full-load operation begins again. Tables with fewer rows than the number set for this option restart the full load. The default value is 1,000,000.
•
ResumeOnlyClusteredPKTables
– Indicates whether to resume loading only tables that have clustered primary keys.
• To indicate the maximum number of tables to load in parallel, set the
MaxFullLoadSubTasks
option.
• To set the number of seconds that AWS DMS waits for transactions to close before beginning a fullload operation, if transactions are open when the task starts, set the
TransactionConsistencyTimeout option. The default value is 600 (10 minutes). AWS DMS begins the full load after the timeout value is reached, even if there are open transactions.
• To indicate the maximum number of events that can be transferred together, set the
CommitRate option.
Logging Task Settings
Logging task settings are written to a JSON file and they let you specify which component activities are logged and what amount of information is written to the log. The logging feature uses Amazon
CloudWatch to log information during the migration process.
There are several ways to enable Amazon CloudWatch logging. You can select the
EnableLogging
option on the AWS Management Console when you create a migration task or set the
EnableLogging
option to true
when creating a task using the AWS DMS API. You can also specify
"EnableLogging": true
in the
JSON of the logging section of task settings.
You can specify logging for the following component activities:
• SOURCE_UNLOAD — Data is unloaded from the source database.
• SOURCE_CAPTURE — Data is captured from the source database.
• TARGET_LOAD — Data is loaded into the target database.
• TARGET_APPLY — Data and DDL are applied to the target database.
• TASK_MANAGER — The task manager triggers an event.
Once you specify a component activity, you can then specify the amount of information that is logged.
The following list is in order from the lowest level of information to the highest level of information. The higher levels always include information from the lower levels. These severity values include:
• LOGGER_SEVERITY_ERROR — Error messages are written to the log.
• LOGGER_SEVERITY_WARNING — Warnings and error messages are written to the log.
API Version API Version 2016-01-01
124
AWS Database Migration Service User Guide
Control Table Task Settings
• LOGGER_SEVERITY_INFO — Informational messages, warnings, and error messages are written to the log.
• LOGGER_SEVERITY_DEFAULT — Debug messages, informational messages, warnings, and error messages are written to the log.
• LOGGER_SEVERITY_DEBUG — Debug messages, informational messages, warnings, and error messages are written to the log.
• LOGGER_SEVERITY_DETAILED_DEBUG — All information is written to the log.
For example, the following JSON section gives task settings for logging for all component activities:
…
"Logging": {
"EnableLogging": true,
"LogComponents": [{
"Id": "SOURCE_UNLOAD",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},{
"Id": "SOURCE_CAPTURE",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},{
"Id": "TARGET_LOAD",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},{
"Id": "TARGET_APPLY",
"Severity": "LOGGER_SEVERITY_INFO"
},{
"Id": "TASK_MANAGER",
"Severity": "LOGGER_SEVERITY_DEBUG"
}]
},
…
Control Table Task Settings
Control tables provide information about the AWS DMS task, as well as useful statistics that you can use to plan and manage both the current migration task and future tasks. You can apply these task settings in a JSON file or using the Advanced Settings link on the Create task page in the AWS DMS console. In addition to the Apply Exceptions (dmslogs.awsdms_apply_exceptions) table, which is always created, you can choose to create additional tables including the following:
• Replication Status (dmslogs.awsdms_status) – This table provides details about the current task including task status, amount of memory consumed by the task, number of changes not yet applied to the target, and the position in the source database from which AWS DMS is currently reading. It also indicates if the task is a full load or change data capture (CDC).
• Suspended Tables (dmslogs.awsdms_suspended_tables) – This table provides a list of suspended tables as well as the reason they were suspended.
• Replication History (dmslogs.awsdms_history) – This table provides information about the replication history including the number and volume of records processed during the task, latency at the end of a CDC task, and other statistics.
The Apply Exceptions (dmslogs.awsdms_apply_exceptions) table contains the following parameters:
Column
TASK_NAME
Type
nvchar
Description
The name of the AWS DMS task.
API Version API Version 2016-01-01
125
AWS Database Migration Service User Guide
Control Table Task Settings
Column
TABLE_OWNER
TABLE_NAME
ERROR_TIME
STATEMENT
ERROR
Type
nvchar nvchar timestamp nvchar nvchar
Description
The table owner.
The table name.
The time the exception (error) occurred.
The statement that was being run when the error occurred.
The error name and description.
The Replication History (dmslogs.awsdms_history) table contains the following parameters:
Column
SERVER_NAME
TASK_NAME
TIMESLOT_TYPE
Type
nvchar nvchar varchar
Description
The name of the machine where the replication task is running.
The name of the AWS DMS task.
One of the following values:
• FULL LOAD
• CHANGE PROCESSING (CDC)
TIMESLOT
TIMESLOT_DURATION
TIMESLOT_LATENCY
RECORDS
TIMESLOT_VOLUME timestamp int int int int
If the task is running both full load and CDC, two history records are written to the time slot.
The ending timestamp of the time slot.
The duration of the time slot.
The target latency at the end of the time slot. This value is only applicable to CDC time slots.
The number of records processed during the time slot.
The volume of data processed in MB.
The Replication Status (dmslogs.awsdms_status) table contains the current status of the task and the target database. It has the following settings:
Column
SERVER_NAME
TASK_NAME
Type
nvchar nvchar
Description
The name of the machine where the replication task is running.
The name of the AWS DMS task.
API Version API Version 2016-01-01
126
AWS Database Migration Service User Guide
Control Table Task Settings
Column
TASK_STATUS
STATUS_TIME
PENDING_CHANGES
DISK_SWAP_SIZE
TASK_MEMORY
SOURCE_CURRENT
_POSITION
SOURCE_CURRENT
_TIMESTAMP
SOURCE_TAIL
_POSITION
SOURCE_TAIL
_TIMESTAMP
SOURCE_TIMESTAMP
_APPLIED
Type
varchar timestamp int int int varchar timestamp varchar timestamp timestamp
Description
One of the following values:
• FULL LOAD
• CHANGE PROCESSING (CDC)
Task status is set to FULL LOAD as long as there is at least one table in full load. After all tables have been loaded, the task status changes to CHANGE PROCESSING if CDC is enabled.
The timestamp of the task status.
The number of change records that were not applied to the target.
The amount of disk space used by old or offloaded transactions.
Current memory used, in MB.
The position in the source database that AWS DMS is currently reading from.
The timestamp in the source database that AWS DMS is currently reading from.
The position of the oldest start transaction that is not committed.
This value is the newest position that you can revert to without losing any changes.
The timestamp of the oldest start transaction that is not committed.
This value is the newest timestamp that you can revert to without losing any changes..
The timestamp of the last transaction commit. In a bulk apply process, this value is the timestamp for the commit of the last transaction in the batch.
Additional control table settings include the following:
•
ControlSchema
– Use this option to indicate the database schema name for the AWS DMS target
Control Tables. If you do not enter any information in this field, then the tables are copied to the default location in the database.
•
HistoryTimeslotInMinutes
– Use this option to indicate the length of each time slot in the Replication
History table. The default is 5 minutes.
API Version API Version 2016-01-01
127
AWS Database Migration Service User Guide
Stream Buffer Task Settings
Stream Buffer Task Settings
You can set stream buffer settings using the AWS CLI, include the following:
•
StreamBufferCount
– Use this option to specify the number of data stream buffers for the migration task. The default stream buffer number is 3. Increasing the value of this setting may increase the speed of data extraction. However, this performance increase is highly dependent on the migration environment, including the source system and instance class of the replication server. The default is sufficient for most situations.
•
StreamBufferSizeInMB
– Use this option to indicate the maximum size of each data stream buffer. The default size is 8 MB. You might need to increase the value for this option when you work with very large LOBs or if you receive a message in the log files stating that the stream buffer size is insufficient.
When calculating the size of this option you can use the following equation: [Max LOB size (or LOB chunk size)]*[number of LOB columns]*[number of stream buffers]*[number of tables loading in parallel per task(MaxFullLoadSubTasks)]*3
•
CtrlStreamBufferSizeInMB
– Use this option to set the size of the control stream buffer. Value is in
MB, and can be from 1 to 8. The default value is 5. You may need to increase this when working with a very large number of tables, such as tens of thousands of tables.
Change Processing Tuning Settings
The following settings determine how AWS DMS handles changes for target tables during change data capture (CDC). Several of these settings depend on the value of the target metadata parameter
BatchApplyEnabled
. For more information on the
BatchApplyEnabled
parameter, see
.
Change processing tuning settings include the following:
The following settings apply only when the target metadata parameter
BatchApplyEnabled
is set to true
.
•
BatchApplyPreserveTransaction
– If set to true
, transactional integrity is preserved and a batch is guaranteed to contain all the changes within a transaction from the source. The default value is true
.
If set to false
, there can be temporary lapses in transactional integrity to improve performance. There is no guarantee that all the changes within a transaction from the source will be applied to the target in a single batch.
•
BatchApplyTimeoutMin
– Sets the minimum amount of time in seconds that AWS DMS waits between each application of batch changes. The default value is 1.
•
BatchApplyTimeoutMax
– Sets the maximum amount of time in seconds that AWS DMS waits between each application of batch changes before timing out. The default value is 30.
•
BatchApplyMemoryLimit
– Sets the maximum amount of memory in (MB) to use for pre-processing in
Batch optimized apply mode. The default value is 500.
•
BatchSplitSize
– Sets the number of changes applied in a single change processing statement. Select the check box and then optionally change the default value. The default value is 10,000. A value of 0 means there is no limit applied.
The following settings apply only when the target metadata parameter
BatchApplyEnabled
is set to false
.
•
MinTransactionSize
– Sets the minimum number of changes to include in each transaction. The default value is 1000.
•
CommitTimeout
– Sets the maximum time in seconds for AWS DMS to collect transactions in batches before declaring a timeout. The default value is 1.
API Version API Version 2016-01-01
128
AWS Database Migration Service User Guide
Change Processing DDL Handling Policy Task Settings
•
HandleSourceTableAltered
– Set this option to true
to alter the target table when the source table is altered.
AWS DMS attempts to keep transaction data in memory until the transaction is fully committed to the source and/or the target. However, transactions that are larger than the allocated memory or that are not committed within the specified time limit are written to disk.
The following settings apply to change processing tuning regardless of the change processing mode.
•
MemoryLimitTotal
– Sets the maximum size (in MB) that all transactions can occupy in memory before being written to disk. The default value is 1024.
•
MemoryKeepTime
– Sets the maximum time in seconds that each transaction can stay in memory before being written to disk. The duration is calculated from the time that AWS DMS started capturing the transaction. The default value is 60.
•
StatementCacheSize
– Sets the maximum number of prepared statements to store on the server for later execution when applying changes to the target. The default value is 50. The maximum value is
200.
Change Processing DDL Handling Policy Task Settings
The following settings determine how AWS DMS handles DDL changes for target tables during change data capture (CDC). Change processing DDL handling policy settings include the following:
•
HandleSourceTableDropped
– Set this option to true
to drop the target table when the source table is dropped
•
HandleSourceTableTruncated
– Set this option to true
to truncate the target table when the source table is truncated
•
HandleSourceTableAltered
– Set this option to true
to alter the target table when the source table is altered.
Error Handling Task Settings
You can set the error handling behavior of your replication task during change data capture (CDC) using the following settings:
•
DataErrorPolicy
– Determines the action AWS DMS takes when there is an error related to data processing at the record level. Some examples of data processing errors include conversion errors, errors in transformation, and bad data. The default is
LOG_ERROR
.
•
IGNORE_RECORD
– The task continues and the data for that record is ignored. The error counter for the
DataErrorEscalationCount
property is incremented so that if you set a limit on errors for a table, this error will count toward that limit.
•
LOG_ERROR
– The task continues and the error is written to the task log.
•
SUSPEND_TABLE
– The task continues but data from the table with the error record is moved into an error state and the data is not replicated.
•
STOP_TASK
– The task stops and manual intervention is required.
•
DataTruncationErrorPolicy
– Determines the action AWS DMS takes when data is truncated. The default is
LOG_ERROR
.
•
IGNORE_RECORD
– The task continues and the data for that record is ignored. The error counter for the
DataErrorEscalationCount
property is incremented so that if you set a limit on errors for a table, this error will count toward that limit.
•
LOG_ERROR
– The task continues and the error is written to the task log.
API Version API Version 2016-01-01
129
AWS Database Migration Service User Guide
Error Handling Task Settings
•
SUSPEND_TABLE
– The task continues but data from the table with the error record is moved into an error state and the data is not replicated.
•
STOP_TASK
– The task stops and manual intervention is required.
•
DataErrorEscalationPolicy
– Determines the action AWS DMS takes when the maximum number of errors (set in the
DataErrorsEscalationCount
parameter) is reached. The default is
SUSPEND_TABLE
.
•
LOG_ERROR
– The task continues and the error is written to the task log.
•
SUSPEND_TABLE
– The task continues but data from the table with the error record is moved into an error state and the data is not replicated.
•
STOP_TASK
– The task stops and manual intervention is required.
•
DataErrorEscalationCount
– Sets the maximum number of errors that can occur to the data for a specific record. When this number is reached, the data for the table that contains the error record is handled according to the policy set in the
DataErrorEscalationCount
. The default is 0.
•
TableErrorPolicy
– Determines the action AWS DMS takes when an error occurs when processing data or metadata for a specific table. This error only applies to general table data and is not an error that relates to a specific record. The default is
SUSPEND_TABLE
.
•
SUSPEND_TABLE
– The task continues but data from the table with the error record is moved into an error state and the data is not replicated.
•
STOP_TASK
– The task stops and manual intervention is required.
•
TableErrorEscalationPolicy
– Determines the action AWS DMS takes when the maximum number of errors (set using the
TableErrorEscalationCount
parameter). The default and only user setting is
STOP_TASK
, where the task is stopped and manual intervention is required.
•
TableErrorEscalationCount
– The maximum number of errors that can occur to the general data or metadata for a specific table. When this number is reached, the data for the table is handled according to the policy set in the
TableErrorEscalationPolicy
. The default is 0.
•
RecoverableErrorCount
– The maximum number of attempts made to restart a task when an environmental error occurs. After the system attempts to restart the task the designated number of times, the task is stopped and manual intervention is required. Set this value to -1 to attempt a restart six times. Set this value to 0 to never attempt to restart a task. The default is 1.
•
RecoverableErrorInterval
– The number of seconds that AWS DMS waits between attempts to restart a task. The default is 5.
•
RecoverableErrorThrottling
– When enabled, the interval between attempts to restart a task is increased each time a restart is attempted. The default is true
.
•
RecoverableErrorThrottlingMax
– The maximum number of seconds that AWS DMS waits between attempts to restart a task if
RecoverableErrorThrottling
is enabled. The default is 1800.
•
ApplyErrorDeletePolicy
– Determines what action AWS DMS takes when there is a conflict with a
DELETE operation. The default is
IGNORE_RECORD
.
•
IGNORE_RECORD
– The task continues and the data for that record is ignored. The error counter for the
ApplyErrorEscalationCount
property is incremented so that if you set a limit on errors for a table, this error will count toward that limit.
•
LOG_ERROR
– The task continues and the error is written to the task log.
•
SUSPEND_TABLE
– The task continues but data from the table with the error record is moved into an error state and the data is not replicated.
•
STOP_TASK
– The task stops and manual intervention is required.
•
ApplyErrorInsertPolicy
– Determines what action AWS DMS takes when there is a conflict with an
INSERT operation. The default is
LOG_ERROR
.
•
IGNORE_RECORD
– The task continues and the data for that record is ignored. The error counter for the
ApplyErrorEscalationCount
property is incremented so that if you set a limit on errors for a table, this error will count toward that limit.
•
LOG_ERROR
– The task continues and the error is written to the task log.
•
SUSPEND_TABLE
– The task continues but data from the table with the error record is moved into an error state and the data is not replicated.
API Version API Version 2016-01-01
130
AWS Database Migration Service User Guide
Saving Task Settings
•
STOP_TASK
– The task stops and manual intervention is required.
•
UPDATE_RECORD
– If there is an existing target record with the same primary key as the inserted source record, the target record is updated.
•
ApplyErrorUpdatePolicy
– Determines what action AWS DMS takes when there is a conflict with an
UPDATE operation. The default is
LOG_ERROR
.
•
IGNORE_RECORD
– The task continues and the data for that record is ignored. The error counter for the
ApplyErrorEscalationCount
property is incremented so that if you set a limit on errors for a table, this error will count toward that limit.
•
LOG_ERROR
– The task continues and the error is written to the task log.
•
SUSPEND_TABLE
– The task continues but data from the table with the error record is moved into an error state and the data is not replicated.
•
STOP_TASK
– The task stops and manual intervention is required.
•
UPDATE_RECORD
– If the target record is missing, the missing target record will be inserted into the target table. Selecting this option requires full supplemental logging to be enabled for all the source table columns when Oracle is the source database.
•
ApplyErrorEscalationPolicy
– Determines what action AWS DMS takes when the maximum number of errors (set using the
ApplyErrorsEscalationCount
parameter) is reached.
•
LOG_ERROR
– The task continues and the error is written to the task log.
•
SUSPEND_TABLE
– The task continues but data from the table with the error record is moved into an error state and the data is not replicated.
•
STOP_TASK
– The task stops and manual intervention is required.
•
ApplyErrorEscalationCount
– Sets the maximum number of APPLY conflicts that can occur for a specific table during a change process operation. When this number is reached, the data for the table is handled according to the policy set in the
ApplyErrorEscalationPolicy
parameter. The default is 0.
•
FulloadIgnoreConflicts
– Determines if AWS DMS loads the conflicting data when carrying out a fullload operation after the change processing is complete.
Saving Task Settings
You can save the settings for a task as a JSON file, in case you want to reuse the settings for another task.
For example, the following JSON file contains settings saved for a task:
{
"TargetMetadata": {
"TargetSchema": "",
"SupportLobs": true,
"FullLobMode": false,
"LobChunkSize": 64,
"LimitedSizeLobMode": true,
"LobMaxSize": 32,
"BatchApplyEnabled": true
},
"FullLoadSettings": {
"FullLoadEnabled": false,
"ApplyChangesEnabled": true,
"TargetTablePrepMode": "DO_NOTHING",
"CreatePkAfterFullLoad": false,
"StopTaskCachedChangesApplied": false,
"StopTaskCachedChangesNotApplied": false,
"ResumeEnabled": false,
"ResumeMinTableSize": 100000,
"ResumeOnlyClusteredPKTables": true,
API Version API Version 2016-01-01
131
AWS Database Migration Service User Guide
Saving Task Settings
"MaxFullLoadSubTasks": 8,
"TransactionConsistencyTimeout": 600,
"CommitRate": 10000
},
"Logging": {
"EnableLogging": false
},
"ControlTablesSettings": {
"ControlSchema":"",
"HistoryTimeslotInMinutes":5,
"HistoryTableEnabled": false,
"SuspendedTablesTableEnabled": false,
"StatusTableEnabled": false
},
"StreamBufferSettings": {
"StreamBufferCount": 3,
"StreamBufferSizeInMB": 8
},
"ChangeProcessingTuning": {
"BatchApplyPreserveTransaction": true,
"BatchApplyTimeoutMin": 1,
"BatchApplyTimeoutMax": 30,
"BatchApplyMemoryLimit": 500,
"BatchSplitSize": 0,
"MinTransactionSize": 1000,
"CommitTimeout": 1,
"MemoryLimitTotal": 1024,
"MemoryKeepTime": 60,
"StatementCacheSize": 50
},
"ChangeProcessingDdlHandlingPolicy": {
"HandleSourceTableDropped": true,
"HandleSourceTableTruncated": true,
"HandleSourceTableAltered": true
},
"ErrorBehavior": {
"DataErrorPolicy": "LOG_ERROR",
"DataTruncationErrorPolicy":"LOG_ERROR",
"DataErrorEscalationPolicy":"SUSPEND_TABLE",
"DataErrorEscalationCount": 50,
"TableErrorPolicy":"SUSPEND_TABLE",
"TableErrorEscalationPolicy":"STOP_TASK",
"TableErrorEscalationCount": 50,
"RecoverableErrorCount": 0,
"RecoverableErrorInterval": 5,
"RecoverableErrorThrottling": true,
"RecoverableErrorThrottlingMax": 1800,
"ApplyErrorDeletePolicy":"IGNORE_RECORD",
"ApplyErrorInsertPolicy":"LOG_ERROR",
"ApplyErrorUpdatePolicy":"LOG_ERROR",
"ApplyErrorEscalationPolicy":"LOG_ERROR",
"ApplyErrorEscalationCount": 0,
"FullLoadIgnoreConflicts": true
}
}
API Version API Version 2016-01-01
132
AWS Database Migration Service User Guide
Table Mapping
Using Table Mapping with an Task to Select and
Filter Data
Table mapping uses several types of rules to specify the data source, source schema, data, and any transformations that should occur during the task. You can use table mapping to specify individual tables in a database to migrate and the schema to use for the migration. In addition, you can use filters to specify what data from a given table column you want replicated and you can use transformations to modify the data written to the target database.
Selection and Transformation Table Mapping using the AWS Console
You can use the AWS console to perform table mapping, including specifying table selection and transformations. In the AWS Console user interface, you use the Where section to specify the schema, table, and action (include or exclude). You use the Filter section to specify the column name in a table and the conditions you want to apply to the replication task. Together these two actions create a selection rule.
Transformations can be included in a table mapping after you have specified at least one selection rule.
Transformations can be used to rename a schema or table, add a prefix or suffix to a schema or table, or remove a table column.
The following example shows how to set up selection rules for a table called Customers in a schema called EntertainmentAgencySample. Note that the Guided tab, where you create selection rules and transformations, only appears when you have a source endpoint that has schema and table information.
To specify a table selection, filter criteria, and transformations using the AWS console
1. Sign in to the AWS Management Console and choose AWS DMS. Note that if you are signed in as an AWS Identity and Access Management (IAM) user, you must have the appropriate permissions to access AWS DMS. For more information on the permissions required, see
.
2. On the Dashboard page, choose Tasks.
3. Select Create Task.
4. Enter the task information, including Task name, Replication instance, Source endpoint, Target
endpoint, and Migration type. Select Guided from the Table mappings section.
API Version API Version 2016-01-01
133
AWS Database Migration Service User Guide
Selection and Transformation Table
Mapping using the AWS Console
5. In the Table mapping section, select the schema name and table name. You can use the "%" as a wildcard value when specifying the table name. Specify the action to be taken, to include or exclude data defined by the filter.
API Version API Version 2016-01-01
134
AWS Database Migration Service User Guide
Selection and Transformation Table
Mapping using the AWS Console
6. You specify filter information using the Add column filter and the Add condition links. First, select
Add column filter to specify a column and conditions. Select Add condition to add additional conditions. The following example shows a filter for the
Customers
table that includes
AgencyIDs
between
01
and
85
.
API Version API Version 2016-01-01
135
AWS Database Migration Service User Guide
Selection and Transformation Table
Mapping using the AWS Console
7. When you have created the selections you want, select Add selection rule.
8. After you have created at least one selection rule, you can add a transformation to the task. Select
add transformation rule.
9. Select the target you want to transform and enter the additional information requested. The following example shows a transformation that deletes the
AgencyStatus
column from the
Customer
table.
API Version API Version 2016-01-01
136
AWS Database Migration Service User Guide
Selection and Transformation Table
Mapping using the AWS Console
10. Choose Add transformation rule.
11. You can add additional selection rules or transformations by selecting the add selection rule or add
transformation rule. When you are finished, choose Create task.
API Version API Version 2016-01-01
137
AWS Database Migration Service User Guide
Selection and Transformation Table Mapping using JSON
Selection and Transformation Table Mapping using
JSON
Table mappings can be created in the JSON format. If you create a migration task using the AWS DMS
Management Console, you can enter the JSON directly into the table mapping box. If you use the AWS
Command Line Interface (AWS CLI) or AWS Database Migration Service API to perform migrations, you can create a JSON file to specify the table mappings that you want to occur during migration.
You can specify what tables or schemas you want to work with, and you can perform schema and table transformations. You create table mapping rules using the selection
and transformation
rule types.
Selection Rules and Actions
Using table mapping, you can specify what tables or schemas you want to work with by using selection rules and actions. For table mapping rules that use the selection rule type, the following values can be applied:
Parameter
rule-type rule-id rule-name rule-action
Possible Values
selection
A numeric value.
An alpha-numeric value.
include
, exclude
Description
You must have at least one selection rule when specifying a table mapping.
A unique numeric value to identify the rule.
A unique name to identify the rule.
Include or exclude the object selected by the rule.
Example Migrate All Tables in a Schema
The following example migrates all tables from a schema named
Test
in your source to your target endpoint:
{
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "Test",
"table-name": "%"
},
"rule-action": "include"
}
]
}
Example Migrate Some Tables in a Schema
The following example migrates all tables except those starting with
DMS
from a schema named
Test
in your source to your target endpoint:
{
"rules": [
API Version API Version 2016-01-01
138
AWS Database Migration Service User Guide
Selection and Transformation Table Mapping using JSON
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "Test",
"table-name": "%"
},
"rule-action": "include"
},
{
"rule-type": "selection",
"rule-id": "2",
"rule-name": "2",
"object-locator": {
"schema-name": "Test",
"table-name": "DMS%"
},
"rule-action": "exclude"
}
]
}
Transformation Rules and Actions
You use the transformation actions to specify any transformations you want to apply to the selected schema or table. Transformation rules are optional.
For table mapping rules that use the transformation rule type, the following values can be applied:
Parameter
rule-type rule-id rule-name object-locator rule-action rule-target value
Possible Values
transformation
A numeric value.
An alpha-numeric value.
schema-name
The name of the schema.
table-name
The name of the table. You can use the "%" percent sign to be a wildcard.
• rename
• remove-column
• convert-lowercase
, convertuppercase
• add-prefix
, remove-prefix
, replace-prefix
• add-suffix
, remove-suffix
, replace-suffix schema
, table
, column
An alpha-numeric value that follows the naming rules for the target type.
Description
Applies a to the object specified by the selection rule.
A unique numeric value to identify the rule.
A unique name to identify the rule.
The schema and table the rule applies to.
The transformation you want to apply to the object. All transformation rule actions are case sensitive.
The type of object you are transforming.
The new value for actions that require input, such as rename
.
API Version API Version 2016-01-01
139
AWS Database Migration Service User Guide
Selection and Transformation Table Mapping using JSON
Parameter
old-value
Possible Values
An alpha-numeric value that follows the naming rules for the target type.
Description
The old value for actions that require replacement, such as replace-prefix
.
Example Rename a Schema
The following example renames a schema from
Test
in your source to
Test1
in your target endpoint:
{
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "Test",
"table-name": "%"
},
"rule-action": "include"
},
{
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-action": "rename",
"rule-target": "schema",
"object-locator": {
"schema-name": "Test"
},
"value": "Test1"
}
]
}
Example Rename a Table
The following example renames a table from
Actor
in your source to
Actor1
in your target endpoint:
{
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "Test",
"table-name": "%"
},
"rule-action": "include"
},
{
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-action": "rename",
"rule-target": "table",
"object-locator": {
"schema-name": "Test",
"table-name": "Actor"
},
API Version API Version 2016-01-01
140
AWS Database Migration Service User Guide
Selection and Transformation Table Mapping using JSON
"value": "Actor1"
}
]
}
Example Rename a Column
The following example renames a column in table
Actor
from
first_name
in your source to
fname
in your target endpoint:
{
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "%"
},
"rule-action": "include"
},
{
"rule-type": "transformation",
"rule-id": "4",
"rule-name": "4",
"rule-action": "rename",
"rule-target": "column",
"object-locator": {
"schema-name": "test",
"table-name": "Actor",
"column-name" : "first_name"
},
"value": "fname"
}
]
}
Example Remove a Column
The following example transforms the table named
Actor
in your source to remove all columns starting with the characters
col
from it in your target endpoint:
{
"rules": [{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "%"
},
"rule-action": "include"
}, {
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-action": "remove-column",
"rule-target": "column",
"object-locator": {
"schema-name": "test",
"table-name": "Actor",
API Version API Version 2016-01-01
141
AWS Database Migration Service User Guide
Selection and Transformation Table Mapping using JSON
"column-name": "col%"
}
}]
}
Example Convert to Lowercase
The following example converts a table name from
ACTOR
in your source to
actor
in your target endpoint:
{
"rules": [{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "%"
},
"rule-action": "include"
}, {
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-action": "convert-lowercase",
"rule-target": "table",
"object-locator": {
"schema-name": "test",
"table-name": "ACTOR"
}
}]
}
Example Convert to Uppercase
The following example converts all columns in all tables and all schemas from lowercase in your source to uppercase in your target endpoint:
{
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "%"
},
"rule-action": "include"
},
{
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-action": "convert-uppercase",
"rule-target": "column",
"object-locator": {
"schema-name": "%",
"table-name": "%",
"column-name": "%"
}
}
API Version API Version 2016-01-01
142
AWS Database Migration Service User Guide
Selection and Transformation Table Mapping using JSON
]
}
Example Add a Prefix
The following example transforms all tables in your source to add the prefix
DMS_
to them in your target endpoint:
{
"rules": [{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "%"
},
"rule-action": "include"
}, {
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-action": "add-prefix",
"rule-target": "table",
"object-locator": {
"schema-name": "test",
"table-name": "%"
},
}
"value": "DMS_"
}]
Example Replace a Prefix
The following example transforms all columns containing the prefix
Pre_
in your source to replace the prefix with
NewPre_
in your target endpoint:
{
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "%"
},
"rule-action": "include"
},
{
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-action": "replace-prefix",
"rule-target": "column",
"object-locator": {
"schema-name": "%",
"table-name": "%",
"column-name": "%"
},
"value": "NewPre_",
"old-value": "Pre_"
API Version API Version 2016-01-01
143
AWS Database Migration Service User Guide
Using Source Filters
}
]
}
Example Remove a Suffix
The following example transforms all tables in your source to remove the suffix
_DMS
from them in your target endpoint:
{
"rules": [{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "%"
},
"rule-action": "include"
}, {
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-action": "remove-suffix",
"rule-target": "table",
"object-locator": {
"schema-name": "test",
"table-name": "%"
},
"value": "_DMS"
}]
}
Using Source Filters
You can use source filters to limit the number and type of records transferred from your source to your target. For example, you can specify that only employees with a location of headquarters are moved to the target database. Filters are part of a selection rule. You apply filters on a column of data.
Source filters must follow these constraints:
• A selection rule can have no filters or one or more filters.
• Every filter can have one or more filter conditions.
• If more than one filter is used, the list of filters will be combined as if using an AND operator between the filters.
• If more than one filter condition is used within a single filter, the list of filter conditions will be combined as if using an OR operator between the filter conditions.
• Filters are only applied when rule-action = 'include'
.
• Filters require a column name and a list of filter conditions. Filter conditions must have a filter operator and a value.
• Column names, table names, and schema names are case sensitive.
Filtering by Time and Date
When selecting data to import, you can specify a date or time as part of your filter criteria. AWS DMS uses the date format YYYY-MM-DD and the time format YYYY-MM-DD HH:MM:SS for filtering. The AWS
API Version API Version 2016-01-01
144
AWS Database Migration Service User Guide
Using Source Filters
DMS comparison functions follow the SQLite conventions. For more information about SQLite data types and date comparisons, see Datatypes In SQLite Version 3 .
The following example shows how to filter on a date.
Example Single Date Filter
The following filter replicates all employees where empstartdate >= January 1, 2002
to the target database.
{
"rules": [{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "employee"
},
"rule-action": "include",
"filters": [{
"filter-type": "source",
"column-name": "empstartdate",
"filter-conditions": [{
"filter-operator": "gte",
"value": "2002-01-01"
}]
}]
}]
}
Creating Source Filter Rules in JSON
You can create source filters by specifying a column name, filter condition, filter operator, and a filter value.
The following table shows the parameters used for source filtering.
Parameter
filter-type column-name
filter-conditions
filter-operator value
Value
source
The name of the source column you want the filter applied to. The name is case sensitive.
This parameter can be one of the following:
• ste
– less than or equal to
• gte
– greater than or equal to
• eq
– equal to
• between
– equal to or between two values
The value of the filter-operator parameter. If the filter-operator is between
, provide two values, one for start-value and one for end-value.
The following examples show some common ways to use source filters.
API Version API Version 2016-01-01
145
AWS Database Migration Service User Guide
Using Source Filters
Example Single Filter
The following filter replicates all employees where empid >= 100
to the target database.
{
"rules": [{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "employee"
},
"rule-action": "include",
"filters": [{
"filter-type": "source",
"column-name": "empid",
"filter-conditions": [{
"filter-operator": "gte",
"value": "100"
}]
}]
}]
}
Example Multiple Filter Operators
The following filter applies multiple filter operators to a single column of data. The filter replicates all employees where
(empid <=10)
OR
(empid is between 50 and 75)
OR
(empid >= 100)
to the target database.
{
"rules": [{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "employee"
},
"rule-action": "include",
"filters": [{
"filter-type": "source",
"column-name": "empid",
"filter-conditions": [{
"filter-operator": "ste",
"value": "10"
}, {
"filter-operator": "between",
"start-value": "50",
"end-value": "75"
}, {
"filter-operator": "gte",
"value": "100"
}]
}]
}]
}
API Version API Version 2016-01-01
146
AWS Database Migration Service User Guide
Using Source Filters
Example Multiple Filters
The following filter applies multiple filters to two columns in a table. The filter replicates all employees where
(empid <= 100)
AND
(dept= tech)
to the target database.
{
"rules": [{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "test",
"table-name": "employee"
},
"rule-action": "include",
"filters": [{
"filter-type": "source",
"column-name": "empid",
"filter-conditions": [{
"filter-operator": "ste",
"value": "100"
}]
}, {
"filter-type": "source",
"column-name": "dept",
"filter-conditions": [{
"filter-operator": "eq",
"value": "tech"
}]
}]
}]
}
API Version API Version 2016-01-01
147
Descarcă
Link public actualizat
Linkul public către chat-ul tău a fost actualizat. You may view and manage your chat links after registration here.
Link public document
Publicitate
