Targets for Data Migration. amazon web services AWS Database Migration Service
Реклама
Реклама
AWS Database Migration Service User Guide
Targets for Data Migration
AWS Database Migration Service (AWS DMS) can use many of the most popular databases as a target for data replication. The target can be on an Amazon Elastic Compute Cloud (Amazon EC2) instance, an
Amazon Relational Database Service (Amazon RDS) instance, or an on-premises database.
Note
Regardless of the source storage engine (MyISAM, MEMORY, etc.), AWS DMS creates the MySQLcompatible target table as an InnoDB table by default. If you need to have a table that uses a storage engine other than InnoDB, you can manually create the table on the MySQL-compatible target and migrate the table using the "Do Nothing" mode. For more information about the "Do
Nothing" mode, see
Full Load Task Settings (p. 123)
.
The databases include the following:
On-premises and EC2 instance databases
• Oracle versions 10g, 11g, 12c, for the Enterprise, Standard, Standard One, and Standard Two editions
• Microsoft SQL Server versions 2005, 2008, 2008R2, 2012, 2014 for the Enterprise, Standard,
Workgroup, and Developer editions. The Web and Express editions are not supported.
• MySQL versions 5.5, 5.6, and 5.7
• MariaDB (supported as a MySQL-compatible data target)
• PostgreSQL versions 9.3 and later
• SAP Adaptive Server Enterprise (ASE) 15.7 and later
Amazon RDS instance databases, Amazon Redshift, Amazon S3, and Amazon DynamoDB
• Amazon RDS Oracle versions 11g (versions 11.2.0.3.v1 and later) and 12c, for the Enterprise, Standard,
Standard One, and Standard Two editions
• Amazon RDS Microsoft SQL Server versions 2008R2, 2012, and 2014, for the Enterprise, Standard,
Workgroup, and Developer editions. The Web and Express editions are not supported.
• Amazon RDS MySQL versions 5.5, 5.6, and 5.7
• Amazon RDS MariaDB (supported as a MySQL-compatible data target)
• Amazon RDS PostgreSQL versions 9.3 and later
• Amazon Aurora
API Version API Version 2016-01-01
94
AWS Database Migration Service User Guide
Using Oracle as a Target
• Amazon Redshift
• Amazon S3
• Amazon DynamoDB
Using an Oracle Database as a Target for AWS
Database Migration Service
You can migrate data to Oracle database targets using AWS DMS, either from another Oracle database or from one of the other supported databases.
You can use SSL to encrypt connections between your Oracle endpoint and the replication instance. For more information on using SSL with an Oracle endpoint, see
Using SSL With AWS Database Migration
.
AWS DMS supports Oracle versions 10g, 11g, and 12c for on-premises and EC2 instances for the
Enterprise, Standard, Standard One, and Standard Two editions as targets. AWS DMS supports Oracle versions 11g (versions 11.2.0.3.v1 and later) and 12c for Amazon RDS instance databases for the
Enterprise, Standard, Standard One, and Standard Two editions.
When using Oracle as a target, we assume the data should be migrated into the schema/user which is used for the target connection. If you want to migrate data to a different schema, you'll need to use a schema transformation to do so. For example, if my target endpoint connects to the user RDSMASTER and you wish to migrate from the user PERFDATA to PERFDATA, you'll need to create a transformation as follows:
{
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-action": "rename",
"rule-target": "schema",
"object-locator": {
"schema-name": "PERFDATA"
},
"value": "PERFDATA"
}
For more information about transformations, see Selection and Transformation Table Mapping using
.
Limitations on Oracle as a Target for AWS Database
Migration Service
Limitations when using Oracle as a target for data migration include the following:
• AWS DMS does not create schema on the target Oracle database. You have to create any schemas you want on the target Oracle database. The schema name must already exist for the Oracle target. Tables from source schema are imported to user/schema, which AWS DMS uses to connect to the target instance. You must create multiple replication tasks if you have to migrate multiple schemas.
• AWS DMS doesn't support the
Use direct path full load
option for tables with INDEXTYPE
CONTEXT. As a workaround, you can use array load.
API Version API Version 2016-01-01
95
AWS Database Migration Service User Guide
User Account Privileges for Oracle as a Target
• In Batch Optimized Apply mode, loading into the net changes table uses Direct Path, which doesn't support XMLType. As a workaround, you can use Transactional Apply mode.
User Account Privileges Required for Using Oracle as a Target
To use an Oracle target in an AWS Database Migration Service task, for the user account specified in the
AWS DMS Oracle database definitions you need to grant the following privileges in the Oracle database:
• SELECT ANY TRANSACTION
• SELECT on V$NLS_PARAMETERS
• SELECT on V$TIMEZONE_NAMES
• SELECT on ALL_INDEXES
• SELECT on ALL_OBJECTS
• SELECT on DBA_OBJECTS
• SELECT on ALL_TABLES
• SELECT on ALL_USERS
• SELECT on ALL_CATALOG
• SELECT on ALL_CONSTRAINTS
• SELECT on ALL_CONS_COLUMNS
• SELECT on ALL_TAB_COLS
• SELECT on ALL_IND_COLUMNS
• DROP ANY TABLE
• SELECT ANY TABLE
• INSERT ANY TABLE
• UPDATE ANY TABLE
• CREATE ANY VIEW
• DROP ANY VIEW
• CREATE ANY PROCEDURE
• ALTER ANY PROCEDURE
• DROP ANY PROCEDURE
• CREATE ANY SEQUENCE
• ALTER ANY SEQUENCE
• DROP ANY SEQUENCE
For the requirements specified following, grant the additional privileges named:
• To use a specific table list, grant SELECT on any replicated table and also ALTER on any replicated table.
• To allow a user to create a table in his default tablespace, grant the privilege GRANT UNLIMITED
TABLESPACE.
• For logon, grant the privilege CREATE SESSION.
• If you are using a direct path, grant the privilege LOCK ANY TABLE.
• If the "DROP and CREATE table" or "TRUNCATE before loading" option is selected in the full load settings, and the target table schema is different from that for the AWS DMS user, grant the privilege
DROP ANY TABLE.
API Version API Version 2016-01-01
96
AWS Database Migration Service User Guide
Configuring Oracle as a Target
• To store changes in change tables or an audit table when the target table schema is different from that for the AWS DMS user, grant the privileges CREATE ANY TABLE and CREATE ANY INDEX.
Read Privileges Required for AWS Database Migration Service on the Target Database
The AWS DMS user account must be granted read permissions for the following DBA tables:
• SELECT on DBA_USERS
• SELECT on DBA_TAB_PRIVS
• SELECT on DBA_OBJECTS
• SELECT on DBA_SYNONYMS
• SELECT on DBA_SEQUENCES
• SELECT on DBA_TYPES
• SELECT on DBA_INDEXES
• SELECT on DBA_TABLES
• SELECT on DBA_TRIGGERS
If any of the required privileges cannot be granted to V$xxx, then grant them to V_$xxx.
Configuring an Oracle Database as a Target for AWS
Database Migration Service
Before using an Oracle database as a data migration target, you must provide an Oracle user account to
AWS DMS. The user account must have read/write privileges on the Oracle database, as specified in the
section User Account Privileges Required for Using Oracle as a Target (p. 96) .
Using a Microsoft SQL Server Database as a Target for AWS Database Migration Service
You can migrate data to Microsoft SQL Server databases using AWS DMS. With an SQL Server database as a target, you can migrate data from either another SQL Server database or one of the other supported databases.
For on-premises and Amazon EC2 instance databases, AWS DMS supports as a target SQL Server versions
2005, 2008, 2008R2, 2012, and 2014, for the Enterprise, Standard, Workgroup, and Developer editions.
The Web and Express editions are not supported.
For Amazon RDS instance databases, AWS DMS supports as a target SQL Server versions 2008R2, 2012, and 2014, for the Enterprise, Standard, Workgroup, and Developer editions are supported. The Web and
Express editions are not supported.
For additional details on working with AWS DMS and SQL Server target databases, see the following.
Topics
•
Limitations on Using SQL Server as a Target for AWS Database Migration Service (p. 98)
•
Security Requirements When Using SQL Server as a Target for AWS Database Migration
API Version API Version 2016-01-01
97
AWS Database Migration Service User Guide
Limitations on Using SQL Server as a
Target for AWS Database Migration Service
Limitations on Using SQL Server as a Target for AWS
Database Migration Service
The following limitations apply when using a SQL Server database as a target for AWS DMS:
• When you manually create a SQL Server target table with a computed column, full load replication is not supported when using the BCP bulk-copy utility. To use full load replication, disable the
Use BCP
for loading tables option in the console's Advanced tab.
• When replicating tables with SQL Server spatial data types (GEOMETRY and GEOGRAPHY), AWS DMS replaces any spatial reference identifier (SRID) that you might have inserted with the default SRID. The default SRID is 0 for GEOMETRY and 4326 for GEOGRAPHY.
Security Requirements When Using SQL Server as a
Target for AWS Database Migration Service
The following describes the security requirements for using AWS DMS with a Microsoft SQL Server target.
• AWS DMS user account must have at least the db_owner user role on the Microsoft SQL Server database you are connecting to.
• A Microsoft SQL Server system administrator must provide this permission to all AWS DMS user accounts.
Using a PostgreSQL Database as a Target for AWS
Database Migration Service
You can migrate data to PostgreSQL databases using AWS DMS, either from another PostgreSQL database or from one of the other supported databases.
PostgreSQL versions 9.3 and later are supported for on-premises, Amazon RDS, and EC2 instance databases.
Limitations on Using PostgreSQL as a Target for AWS
Database Migration Service
The following limitations apply when using a PostgreSQL database as a target for AWS DMS:
• The JSON data type is converted to the Native CLOB data type.
Security Requirements When Using a PostgreSQL
Database as a Target for AWS Database Migration
Service
For security purposes, the user account used for the data migration must be a registered user in any
PostgreSQL database that you use as a target.
API Version API Version 2016-01-01
98
AWS Database Migration Service User Guide
Using MySQL as a Target
Using a MySQL-Compatible Database as a Target for AWS Database Migration Service
You can migrate data to MySQL databases using AWS DMS, either from another MySQL database or from one of the other supported databases.
MySQL versions 5.5, 5.6, and 5.7, as well as MariaDB and Amazon Aurora, are supported.
Prerequisites for Using a MySQL-Compatible
Database as a Target for AWS Database Migration
Service
Before you begin to work with a MySQL database as a target for AWS DMS, make sure that you have the following prerequisites:
• A MySQL account with the required security settings. For more information, see
When Using MySQL as a Target for AWS Database Migration Service (p. 100) .
• A MySQL database with the tables that you want to replicate accessible in your network. AWS DMS supports the following MySQL editions:
• MySQL Community Edition
• MySQL Standard Edition
• MySQL Enterprise Edition
• MySQL Cluster Carrier Grade Edition
• MariaDB
• Amazon Aurora
• During a load, you should consider disabling foreign keys. In order to disable foreign key checks on a MySQL-compatible database during a load, you can add the following command to the
Extra
Connection Attributes in the Advanced section of the target MySQL, Aurora, MariaDB endpoint connection information: initstmt=SET FOREIGN_KEY_CHECKS=0
Limitations on Using MySQL as a Target for AWS
Database Migration Service
When using a MySQL database as a target, AWS DMS doesn't support the following:
• The DDL statements Truncate Partition, Drop Table, and Rename Table.
• Using an
ALTER TABLE
<table_name>
ADD COLUMN
<column_name>
statement to add columns to the beginning or the middle of a table.
In addition, when you update a column's value to its existing value, MySQL returns a
0 rows affected warning. In contrast, Oracle performs an update of one row in this case. The MySQL result generates an entry in the awsdms_apply_exceptions control table and the following warning:
API Version API Version 2016-01-01
99
AWS Database Migration Service User Guide
Security Requirements for MySQL as a Target
Some changes from the source database had no impact when applied to the target database. See awsdms_apply_exceptions table for details.
Security Requirements When Using MySQL as a
Target for AWS Database Migration Service
When using MySQL as a target for data migration, you must provide MySQL account access to the AWS
DMS user account. This user must have read/write privileges in the MySQL database.
To create the necessary privileges, run the following commands:
CREATE USER '<user acct>'@'%' IDENTIFIED BY <user password>';
GRANT ALTER, CREATE, DROP, INDEX, INSERT, UPDATE, DELETE, SELECT ON myschema.* TO '<user
acct>'@'%';
GRANT ALL PRIVILEGES ON awsdms_control.* TO '<user acct>'@'%';
Using an Amazon Redshift Database as a Target for AWS Database Migration Service
You can migrate data to Amazon Redshift databases using AWS Database Migration Service. Amazon
Redshift is a fully-managed, petabyte-scale data warehouse service in the cloud. With an Amazon
Redshift database as a target, you can migrate data from all of the other supported source databases.
The Amazon Redshift cluster must be in the same AWS account and same AWS Region as the replication instance.
During a database migration to Amazon Redshift, AWS DMS first moves data to an S3 bucket. Once the files reside in an S3 bucket, AWS DMS then transfers them to the proper tables in the Amazon Redshift data warehouse. AWS DMS creates the S3 bucket in the same AWS Region as the Amazon Redshift database. The AWS DMS replication instance must be located in that same region.
If you use the AWS Command Line Interface (AWS CLI) or the AWS DMS API to migrate data to Amazon
Redshift, you must set up an AWS Identity and Access Management (IAM) role to allow S3 access. For more information about creating this IAM role, see
Creating the IAM Roles to Use With the AWS CLI and
The Amazon Redshift endpoint provides full automation for the following:
• Schema generation and data type mapping
• Full load of source database tables
• Incremental load of changes made to source tables
• Application of schema changes in data definition language (DDL) made to the source tables
• Synchronization between full load and change data capture (CDC) processes.
AWS Database Migration Service supports both full load and change processing operations. AWS DMS reads the data from the source database and creates a series of comma-separated value (CSV) files.
API Version API Version 2016-01-01
100
AWS Database Migration Service User Guide
Prerequisites for Using an Amazon Redshift Database as a Target for AWS Database Migration Service
For full-load operations, AWS DMS creates files for each table. AWS DMS then copies the table files for each table to a separate folder in Amazon S3. When the files are uploaded to Amazon S3, AWS DMS sends a copy command and the data in the files are copied into Amazon Redshift. For change-processing operations, AWS DMS copies the net changes to the CSV files. AWS DMS then uploads the net change files to Amazon S3 and copies the data to Amazon Redshift.
Topics
•
Prerequisites for Using an Amazon Redshift Database as a Target for AWS Database Migration
•
Limitations on Using Redshift as a Target for AWS Database Migration Service (p. 101)
•
Configuring an Amazon Redshift Database as a Target for AWS Database Migration Service (p. 102)
•
Using Enhanced VPC Routing with an Amazon Redshift as a Target for AWS Database Migration
Prerequisites for Using an Amazon Redshift Database as a Target for AWS Database Migration Service
The following list describe the prerequisites necessary for working with Amazon Redshift as a target for data migration:
• Use the AWS Management Console to launch an Amazon Redshift cluster. You should note the basic information about your AWS account and your Amazon Redshift cluster, such as your password, user name, and database name. You will need these values when creating the Amazon Redshift target endpoint.
• The Amazon Redshift cluster must be in the same AWS account and the same AWS Region as the replication instance.
• The AWS DMS replication instance needs network connectivity to the Redshift endpoint (hostname and port) that your cluster uses.
• AWS DMS uses an Amazon S3 bucket to transfer data to the Redshift database. For AWS DMS to create the bucket, the DMS console uses an Amazon IAM role, dms-access-for-endpoint
. If you use the AWS
CLI or DMS API to create a database migration with Amazon Redshift as the target database, you must
With the AWS CLI and AWS DMS API (p. 44) .
Limitations on Using Redshift as a Target for AWS
Database Migration Service
When using a Redshift database as a target, AWS DMS doesn't support the following:
• When migrating from MySQL/Aurora to Redshift, you cannot use DDL to alter a column from the
BLOB data type to the NVARCHAR data type.
For example, the following DDL is not supported.
ALTER TABLE table_name MODIFY column_name NVARCHAR(n);
API Version API Version 2016-01-01
101
AWS Database Migration Service User Guide
Configuring an Amazon Redshift Database as a Target for AWS Database Migration Service
Configuring an Amazon Redshift Database as a
Target for AWS Database Migration Service
AWS Database Migration Service must be configured to work with the Amazon Redshift instance. The following table describes the configuration properties available for the Amazon Redshift endpoint.
Property
server port username password database
Description
The name of the Amazon Redshift cluster you are using.
The port number for Amazon Redshift. The default value is 5439.
An Amazon Redshift user name for a registered user.
The password for the user named in the username property.
The name of the Amazon Redshift data warehouse (service) you are working with.
If you want to add extra connection string attributes to your Amazon Redshift endpoint, you can specify the maxFileSize
and fileTransferUploadStreams
attributes. For more information on these attributes,
.
Using Enhanced VPC Routing with an Amazon
Redshift as a Target for AWS Database Migration
Service
If you're using the Enhanced VPC Routing feature with your Amazon Redshift target, the feature forces all COPY traffic between your Redshift cluster and your data repositories through your Amazon VPC.
Because Enhanced VPC Routing affects the way that Amazon Redshift accesses other resources, COPY commands might fail if you haven't configured your VPC correctly.
AWS DMS can be affected by this behavior since it uses the COPY command to move data in S3 to a
Redshift cluster.
Following are the steps AWS DMS takes to load data into an Amazon Redshift target:
1. AWS DMS copies data from the source to CSV files on the replication server.
2. AWS DMS uses the AWS SDK to copy the CSV files into an S3 bucket on your account.
3. AWS DMS then uses the COPY command in Redshift to copy data from the CSV files in S3 to an appropriate table in Redshift.
If Enhanced VPC Routing is not enabled, Amazon Redshift routes traffic through the Internet, including traffic to other services within the AWS network. If the feature is not enabled, you do not have to configure the network path. If the feature is enabled, you must specifically create a network path between your cluster's VPC and your data resources. For more information on the configuration required, see Enhanced VPC Routing in the Amazon Redshift documentation.
API Version API Version 2016-01-01
102
AWS Database Migration Service User Guide
Using SAP ASE as a Target
Using a SAP ASE Database as a Target for AWS
Database Migration Service
You can migrate data to SAP Adaptive Server Enterprise (ASE)–formerly known as Sybase–databases using AWS DMS, either from any of the supported database sources.
SAP ASE version 15.7 and later are supported.
Prerequisites for Using a SAP ASE Database as a
Target for AWS Database Migration Service
Before you begin to work with a SAP ASE database as a target for AWS DMS, make sure that you have the following prerequisites:
• You must provide SAP ASE account access to the AWS DMS user. This user must have read/write privileges in the SAP ASE database.
• When replicating to SAP ASE version 15.7 installed on a Windows EC2 instance configured with a non-
Latin language (for example, Chinese), AWS DMS requires SAP ASE 15.7 SP121 to be installed on the target SAP ASE machine.
Using Amazon S3 as a Target for AWS Database
Migration Service
You can migrate data to Amazon S3 using AWS DMS from any of the supported database sources.
When using S3 as a target in an AWS DMS task, both full load and change data capture (CDC) data is written to comma-separated-values (CSV) format. AWS DMS names files created during a full load using incremental counters, for example LOAD00001.csv, LOAD00002, and so on. AWS DMSnames CDC files using timestamps, for example 20141029-1134010000.csv. For each source table, AWS DMS creates a folder under the specified target folder. AWS DMS writes all full load and CDC files to the specified S3 bucket.
The parameter bucketFolder
contains the location where the .csv files are stored before being uploaded to the S3 bucket. Table data is stored in the following format in the S3 bucket:
<schema_name>/<table_name>/LOAD001.csv
<schema_name>/<table_name>/LOAD002.csv
<schema_name>/<table_name>/<time-stamp>.csv
{
You can specify the column delimiter, row delimiter, null value indicator, and other parameters using the extra connection attributes. For more information on the extra connection attributes, see
Connection Attributes (p. 104) at the end of this section.
Prerequisites for Using Amazon S3 as a Target
The AWS account you use for the migration must have write and delete access to the Amazon S3 bucket you are using as a target. The role assigned to the user account used to create the migration task must have the following set of permissions.
API Version API Version 2016-01-01
103
AWS Database Migration Service User Guide
Limitations to Using Amazon S3 as a Target
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::buckettest2*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::buckettest2*"
]
}
]
}
Limitations to Using Amazon S3 as a Target
The following limitations apply to a file in Amazon S3 that you are using as a target:
• Only the following data definition language (DDL) commands are supported: TRUNCATE TABLE, DROP
TABLE, and CREATE TABLE.
• Full LOB mode is not supported.
• UPDATE and DELETE statements are not supported.
• Changes to the source table structure during full load are not supported. Changes to the data is supported during full load.
Security
To use Amazon S3 as a target, the account used for the migration must have write and delete access to the Amazon S3 bucket that is used as the target. You must specify the Amazon Resource Name (ARN) of an IAM role that has the permissions required to access Amazon S3.
Extra Connection Attributes
You can specify the following options as extra connection attributes.
Option
csvRowDelimiter csvDelimiter
Description
The delimiter used to separate rows in the source files. The default is a carriage return (
\n
).
Example:
csvRowDelimiter=\n;
The delimiter used to separate columns in the source files. The default is a comma.
API Version API Version 2016-01-01
104
AWS Database Migration Service User Guide
Using Amazon DynamoDB as a Target
Option
bucketFolder bucketName compressionType
Description
Example:
csvDelimiter=,;
An optional parameter to set a folder name in the S3 bucket. If provided, tables are created in the path <bucketFolder>/<schema_name>/
<table_name>/. If this parameter is not specified, then the path used is
<schema_name>/<table_name>/.
Example:
bucketFolder=testFolder;
The name of the S3 bucket.
Example:
bucketName=buckettest;
An optional parameter to use GZIP to compress the target files. Set to NONE
(the default) or do not use to leave the files uncompressed.
Example:
compressionType=GZIP;
Using an Amazon DynamoDB Database as a Target for AWS Database Migration Service
You can use AWS DMS to migrate data to an Amazon DynamoDB table. Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. AWS DMS supports using a relational database or MongoDB as a source.
In DynamoDB, tables, items, and attributes are the core components that you work with. A table is a collection of items, and each item is a collection of attributes. DynamoDB uses primary keys, called partition keys, to uniquely identify each item in a table. You can also use keys and secondary indexes to provide more querying flexibility.
You use object mapping to migrate your data from a source database to a target DynamoDB table.
Object mapping lets you determine where the source data is located in the target.
Migrating from a Relational Database to a DynamoDB
Table
AWS DMS supports migrating data to DynamoDB’s scalar data types. When migrating from a relational database like Oracle or MySQL to DynamoDB, you may want to restructure how you store this data.
Currently AWS DMS supports single table to single table restructuring to DynamoDB scalar type attributes. If you are migrating data into DynamoDB from a relational database table, you would take data from a table and reformat it into DynamoDB scalar data type attributes. These attributes can accept data from multiple columns and you can map a column to an attribute directly.
AWS DMS supports the following DynamoDB scalar data types:
API Version API Version 2016-01-01
105
AWS Database Migration Service User Guide
Prerequisites for DynamoDB as a Target
• String
• Number
• Boolean
Note
NULL data from the source are ignored on the target.
Prerequisites for Using a DynamoDB as a Target for
AWS Database Migration Service
Before you begin to work with a DynamoDB database as a target for AWS DMS, make sure you create an IAM role that allows AWS DMS to assume and grants access to the DynamoDB tables that are being migrated into. The minimum set of access permissions is shown in the following sample role policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "dms.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
The user account that you use for the migration to DynamoDB must have the following permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:PutItem",
"dynamodb:CreateTable",
"dynamodb:DescribeTable",
"dynamodb:DeleteTable",
"dynamodb:DeleteItem"
],
"Resource": [
"arn:aws:dynamodb:us-west-2:account-id:table/Name1",
"arn:aws:dynamodb:us-west-2:account-id:table/OtherName*",
]
},
{
"Effect": "Allow",
"Action": [
"dynamodb:ListTables",
],
"Resource": "*"
}
]
}
API Version API Version 2016-01-01
106
AWS Database Migration Service User Guide
Limitations for DynamoDB as a Target
Limitations When Using DynamoDB as a Target for
AWS Database Migration Service
The following limitations apply when using Amazon DynamoDB as a target:
• DynamoDB limits the precision of the Number data type to 38 places. Store all data types with a higher precision as a String. You need to explicitly specify this using the object mapping feature.
• Because Amazon DynamoDB doesn’t have a Date data type, data using the Date data type are converted to strings.
• AWS DMS only supports replication of tables with non-composite primary keys, unless you specify an object mapping for the target table with a custom hash or range key, or both.
• AWS DMS doesn’t support LOB data unless it is a CLOB. AWS DMS converts CLOB data into a
DynamoDB string when migrating data.
Using Object Mapping to Migrate Data to DynamoDB
AWS DMS uses table mapping rules to map data from the source to the target DynamoDB table. To map data to a DynamoDB target, you use a type of table mapping rule called object-mapping. Object mapping lets you define the attribute names and the data to be migrated to them. Amazon DynamoDB doesn't have a preset structure other than having a partition key and an optional sort key. If you have a noncomposite primary key, AWS DMS will use it. If you have a composite primary key or you want to use a sort key, you need to define these keys and the other attributes in your target DynamoDB table.
To create an object mapping rule, you specify the rule-type
as object-mapping. This rule specifies what type of object mapping you want to use.
The structure for the rule is as follows:
{ "rules": [
{
"rule-type": "object-mapping",
"rule-id": "<id>",
"rule-name": "<name>",
"rule-action": "<valid object-mapping rule action>",
"object-locator": {
"schema-name": "<case-sensitive schema name>",
"table-name": ""
},
"target-table-name": "<table_name>",
}
}
]
}
AWS DMS currently supports map-record-to-record and map-record-to-document as the only valid values for the rule-action
parameter. map-record-to-record and map-record-to-document specify what AWS
DMS does by default to records that aren’t excluded as part of the exclude-columns
attribute list; these values don’t affect the attribute-mappings in any way.
• map-record-to-record can be used when migrating from a relational database to DynamoDB. It uses the primary key from the relational database as the partition key in Amazon DynamoDB and creates an attribute for each column in the source database. When using map-record-to-record
, for any column in the source table not listed in the exclude-columns
attribute list, AWS DMS creates a corresponding attribute on the target DynamoDB instance regardless of whether that source column is used in an attribute mapping.
API Version API Version 2016-01-01
107
AWS Database Migration Service User Guide
Using Object Mapping with DynamoDB
• map-record-to-document puts source columns into a single, flat, DynamoDB map on the target using the attribute name "_doc." When using map-record-to-document
, for any column in the source table not listed in the exclude-columns
attribute list, AWS DMS places the data into a single, flat,
DynamoDB map attribute on the source called “_doc”.
One way to understand the difference between the rule-action
parameters map-record-to-record and map-record-to-document is to see the two parameters in action. For this example, assume you are starting with a relational database table row with the following structure and data:
To migrate this information to DynamoDB, you would create rules to map the data into a DynamoDB table item. Note the columns listed for the exclude-columns
parameter. These columns are not directly mapped over to the target; instead, attribute mapping is used to combine the data into new items, such as where FirstName and LastName are grouped together to become CustomerName on the DynamoDB target. Note also that NickName and income are not excluded.
{
"rules": [
{
"rule-type": "object-mapping",
"rule-id": "1",
"rule-name": "TransformToDDB",
"rule-action": "map-record-to-record",
"object-locator": {
"schema-name": "test",
"table-name": "customer",
},
"target-table-name": "customer_t",
"mapping-parameters": {
"partition-key-name": "CustomerName",
"exclude-columns": [
"FirstName",
"LastName",
"HomeAddress",
"HomePhone",
"WorkAddress",
"WorkPhone"
],
"attribute-mappings": [
{
"target-attribute-name": "CustomerName",
"attribute-type": "scalar",
"attribute-sub-type": "string",
"value": "${FirstName},${LastName}"
},
{
"target-attribute-name": "ContactDetails",
"attribute-type": "document",
"attribute-sub-type": "dynamodb-map",
"value": {
"M": {
“Home": {
"M": {
"Address": {
"S": "${HomeAddress}"
},
"Phone": {
"S": "${HomePhone}"
API Version API Version 2016-01-01
108
AWS Database Migration Service User Guide
Using Object Mapping with DynamoDB
}
}
},
"Work": {
"M": {
"Address": {
"S": "${WorkAddress}"
},
"Phone": {
"S": "${WorkPhone}"
}
}
}
}
}
}
]
}
}
]
}
By using the rule-action
parameter map-record-to-record, the data for NickName and income are mapped to items of the same name in the DynamoDB target.
If you use the exact same rules but change the rule-action
parameter to map-record-to-document, the columns not listed in the exclude-columns
parameter, NickName and income, are mapped to a _doc item.
API Version API Version 2016-01-01
109
AWS Database Migration Service User Guide
Using Object Mapping with DynamoDB
Using Custom Condition Expressions with Object Mapping
You can use a feature of Amazon DynamoDB called conditional expressions to manipulate data that is being written to a DynamoDB table. For more information about condition expressions in DynamoDB, see Condition Expressions .
A condition expression member consists of:
• an expression (required)
• expression attribute values (optional) . Specifies a DynamoDB json structure of the attribute value
• expression attribute names (optional)
• options for when to use the condition expression (optional). The default is apply-during-cdc = false and apply-during-full-load = true
The structure for the rule is as follows:
"target-table-name": "customer_t",
"mapping-parameters": {
"partition-key-name": "CustomerName",
"condition-expression": {
"expression":"<conditional expression>",
"expression-attribute-values": [
{
"name":"<attribute name>",
"value":<attribute value>
}
],
"apply-during-cdc":<optional Boolean value>,
"apply-during-full-load": <optional Boolean value>
}
The following sample highlights the sections used for condition expression.
API Version API Version 2016-01-01
110
AWS Database Migration Service User Guide
Using Object Mapping with DynamoDB
Using Attribute Mapping with Object Mapping
Attribute mapping lets you specify a template string using source column names to restructure data on the target. There is no formatting done other than what the user specifies in the template.
The following example shows the structure of the source database and the desired structure of the
DynamoDB target. First is shown the structure of the source, in this case an Oracle database, and then the desired structure of the data in DynamoDB. The example concludes with the JSON used to create the desired target structure.
The structure of the Oracle data is as follows:
Oracle
FirstName LastName StoreId
PrimaryKey
Randy Marsh
N/A
5
HomeAddressHomePhone WorkAddressWorkPhone DateOfBirth
221B
Baker
Street
123456789031
Spooner
Street,
Quahog
987654321002/29/1988
API Version API Version 2016-01-01
111
AWS Database Migration Service User Guide
Using Object Mapping with DynamoDB
The structure of the DynamoDB data is as follows:
DynamoDB
CustomerName
HashKey
Randy,Marsh
StoreId
SortKey
5
ContactDetails
{
N/A
"Name":Randy,
"Home":{
"Address":"221B Baker
Street",
"Phone":1234567890
},
"Work":{
"Address":"31 Spooner
Street, Quahog",
}
}
"Phone":9876541230
DateOfBirth
02/29/1988
The following JSON shows the object mapping and column mapping used to achieve the DynamoDB structure:
{
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "Test",
"table-name": "%"
},
"rule-action": "include"
}
]
{
{
"rule-type": "object-mapping",
"rule-id": "1",
"rule-name": "TransformToDDB",
"rule-action": "map-record-to-record",
"object-locator": {
"owner-name": "test",
"table-name": "customer",
},
"target-table-name": "customer_t",
"mapping-parameters": {
API Version API Version 2016-01-01
112
AWS Database Migration Service User Guide
Using Object Mapping with DynamoDB
"partition-key-name": "CustomerName",
"sort-key-name": "StoreId",
"exclude-columns": [
"FirstName", "LastName", "HomeAddress", "HomePhone", "WorkAddress", "WorkPhone"
],
"attribute-mappings": [
{
"target-attribute-name": "CustomerName",
"attribute-type": "scalar",
"attribute-sub-type": "string",
"value":
"${FirstName},${LastName}"
},
{
"target-attribute-name": "StoreId",
"attribute-type": "scalar",
"attribute-sub-type": "string",
"value":
"${StoreId}"
},
{
"target-attribute-name": "ContactDetails",
"attribute-type": "scalar",
"attribute-sub-type": "string",
"value":
"{
\"Name\":\"${FirstName}\",
\"Home\":{
\"Address\":\"${HomeAddress}\",
\"Phone\":\"${HomePhone}\"
},
\"Work\":{
\"Address\":\"${WorkAddress}\",
\"Phone\":\"${WorkPhone}\"
}
}”
},
{
"target-attribute-name": "StoreId",
"attribute-type": "scalar",
"attribute-sub-type": "string",
"value": {
"${StoreId}"
}
}
]
}
}
]
}
Another way to use column mapping is to use DynamoDB format as your document type. The following code example uses dynamodb-map as the attribute-sub-type
for attribute mapping.
{
"rules": [
{
"rule-type": "object-mapping",
"rule-id": "1",
"rule-name": "TransformToDDB",
"rule-action": "map-record-to-record",
API Version API Version 2016-01-01
113
AWS Database Migration Service User Guide
Using Object Mapping with DynamoDB
"object-locator": {
"schema-name": "test",
"table-name": "customer",
},
"target-table-name": "customer_t",
"mapping-parameters": {
"partition-key-name": "CustomerName",
"sort-key-name": "StoreId",
"exclude-columns": [
"FirstName",
"LastName",
"HomeAddress",
"HomePhone",
"WorkAddress",
"WorkPhone"
],
"attribute-mappings": [
{
"target-attribute-name": "CustomerName",
"attribute-type": "scalar",
"attribute-sub-type": "string",
"value": "${FirstName},${LastName}"
},
{
"target-attribute-name": "StoreId",
"attribute-type": "scalar",
"attribute-sub-type": "string",
"value": "${StoreId}"
},
{
"target-attribute-name": "ContactDetails",
"attribute-type": "document",
"attribute-sub-type": "dynamodb-map",
"value": {
"M": {
"Name": {
"S": "${FirstName}"
}"Home": {
"M": {
"Address": {
"S": "${HomeAddress}"
},
"Phone": {
"S": "${HomePhone}"
}
}
},
"Work": {
"M": {
"Address": {
"S": "${WorkAddress}"
},
"Phone": {
"S": "${WorkPhone}"
}
}
}
}
}
}
]
}
}
]
}
API Version API Version 2016-01-01
114
AWS Database Migration Service User Guide
Using Object Mapping with DynamoDB
Example 1: Using Attribute Mapping with Object Mapping
The following example migrates data from two MySQL database tables, nfl_data and sport_team , to two
DynamoDB table called NFLTeams and SportTeams. The structure of the tables and the JSON used to map the data from the MySQL database tables to the DynamoDB tables are shown following.
The structure of the MySQL database table nfl_data is shown below: mysql> desc nfl_data;
+---------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+-------------+------+-----+---------+-------+
| Position | varchar(5) | YES | | NULL | |
| player_number | smallint(6) | YES | | NULL | |
| Name | varchar(40) | YES | | NULL | |
| status | varchar(10) | YES | | NULL | |
| stat1 | varchar(10) | YES | | NULL | |
| stat1_val | varchar(10) | YES | | NULL | |
| stat2 | varchar(10) | YES | | NULL | |
| stat2_val | varchar(10) | YES | | NULL | |
| stat3 | varchar(10) | YES | | NULL | |
| stat3_val | varchar(10) | YES | | NULL | |
| stat4 | varchar(10) | YES | | NULL | |
| stat4_val | varchar(10) | YES | | NULL | |
| team | varchar(10) | YES | | NULL | |
+---------------+-------------+------+-----+---------+-------+
The structure of the MySQL database table sport_team is shown below: mysql> desc sport_team;
+---------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------------+--------------+------+-----+---------+----------------+
| id | mediumint(9) | NO | PRI | NULL | auto_increment |
| name | varchar(30) | NO | | NULL | |
| abbreviated_name | varchar(10) | YES | | NULL | |
| home_field_id | smallint(6) | YES | MUL | NULL | |
| sport_type_name | varchar(15) | NO | MUL | NULL | |
| sport_league_short_name | varchar(10) | NO | | NULL | |
| sport_division_short_name | varchar(10) | YES | | NULL | |
The table mapping rules used to map the two tables to the two DynamoDB tables is shown below:
{
"rules":[
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "dms_sample",
"table-name": "nfl_data"
API Version API Version 2016-01-01
115
AWS Database Migration Service User Guide
Using Object Mapping with DynamoDB
},
"rule-action": "include"
},
{
"rule-type": "selection",
"rule-id": "2",
"rule-name": "2",
"object-locator": {
"schema-name": "dms_sample",
"table-name": "sport_team"
},
"rule-action": "include"
},
{
"rule-type":"object-mapping",
"rule-id":"3",
"rule-name":"MapNFLData",
"rule-action":"map-record-to-record",
"object-locator":{
"schema-name":"dms_sample",
"table-name":"nfl_data"
},
"target-table-name":"NFLTeams",
"mapping-parameters":{
"partition-key-name":"Team",
"sort-key-name":"PlayerName",
"exclude-columns": [
"player_number", "team", "Name"
],
"attribute-mappings":[
{
"target-attribute-name":"Team",
"attribute-type":"scalar",
"attribute-sub-type":"string",
"value":"${team}"
},
{
"target-attribute-name":"PlayerName",
"attribute-type":"scalar",
"attribute-sub-type":"string",
"value":"${Name}"
},
{
"target-attribute-name":"PlayerInfo",
"attribute-type":"scalar",
"attribute-sub-type":"string",
"value":"{\"Number\": \"${player_number}\",\"Position\": \"${Position}\",
\"Status\": \"${status}\",\"Stats\": {\"Stat1\": \"${stat1}:${stat1_val}\",\"Stat2\":
\"${stat2}:${stat2_val}\",\"Stat3\": \"${stat3}:${ stat3_val}\",\"Stat4\": \"${stat4}:${stat4_val}\"}"
}
]
}
},
{
"rule-type":"object-mapping",
"rule-id":"4",
"rule-name":"MapSportTeam",
"rule-action":"map-record-to-record",
"object-locator":{
"schema-name":"dms_sample",
"table-name":"sport_team"
},
"target-table-name":"SportTeams",
"mapping-parameters":{
"partition-key-name":"TeamName",
API Version API Version 2016-01-01
116
AWS Database Migration Service User Guide
Using Object Mapping with DynamoDB
"exclude-columns": [
"name", "id"
],
"attribute-mappings":[
{
"target-attribute-name":"TeamName",
"attribute-type":"scalar",
"attribute-sub-type":"string",
"value":"${name}"
},
{
"target-attribute-name":"TeamInfo",
"attribute-type":"scalar",
"attribute-sub-type":"string",
"value":"{\"League\": \"${sport_league_short_name}\",\"Division\":
\"${sport_division_short_name}\"}"
}
]
}
}
]
}
The sample output for the NFLTeams DynamoDB table is shown below:
"PlayerInfo": "{\"Number\": \"6\",\"Position\": \"P\",\"Status\": \"ACT\",\"Stats\":
{\"Stat1\": \"PUNTS:73\",\"Stat2\": \"AVG:46\",\"Stat3\": \"LNG:67\",\"Stat4\": \"IN
20:31\"}",
"PlayerName": "Allen, Ryan",
"Position": "P",
"stat1": "PUNTS",
"stat1_val": "73",
"stat2": "AVG",
"stat2_val": "46",
"stat3": "LNG",
"stat3_val": "67",
"stat4": "IN 20",
"stat4_val": "31",
"status": "ACT",
"Team": "NE"
}
The sample output for the SportsTeams DynamoDB table is shown below:
{
"abbreviated_name": "IND",
"home_field_id": 53,
"sport_division_short_name": "AFC South",
"sport_league_short_name": "NFL",
"sport_type_name": "football",
"TeamInfo": "{\"League\": \"NFL\",\"Division\": \"AFC South\"}",
"TeamName": "Indianapolis Colts"
}
API Version API Version 2016-01-01
117
Изтегляне
Публичната връзка е актуализирана
Публичната връзка към вашия чат е актуализирана. You may view and manage your chat links after registration here.
Публична връзка към документа
Реклама
