Sources for Data Migration. amazon web services AWS Database Migration Service

PDF

Έγγραφο

Διαφήμιση

AWS Database Migration Service User Guide

Sources for Data Migration

AWS Database Migration Service (AWS DMS) can use many of the most popular databases as a source for data replication. The source can be on an Amazon Elastic Compute Cloud (Amazon EC2) instance, an

Amazon Relational Database Service (Amazon RDS) instance, or an on-premises database. The source databases include the following.

On-premises and Amazon EC2 instance databases

• Oracle versions 10.2 and later, 11g, and up to 12.1, for the Enterprise, Standard, Standard One, and

Standard Two editions

• Microsoft SQL Server versions 2005, 2008, 2008R2, 2012, and 2014, for the Enterprise, Standard,

Workgroup, and Developer editions. The Web and Express editions are not supported.

• MySQL versions 5.5, 5.6, and 5.7

• MariaDB (supported as a MySQL-compatible data source)

• PostgreSQL 9.3 and later

• SAP Adaptive Server Enterprise (ASE) 15.7 and later

• MongoDB versions 2.6.x and 3.x and later

Amazon RDS instance databases

• Oracle versions 11g (versions 11.2.0.3.v1 and later), and 12c, for the Enterprise, Standard, Standard

One, and Standard Two editions

• Microsoft SQL Server versions 2008R2, 2012, and 2014, for the Enterprise and Standard editions. Note that change data capture (CDC) operations are not supported. The Web, Workgroup, Developer, and

Express editions are not supported.

• MySQL versions 5.5, 5.6, and 5.7. Note that Change Data Capture (CDC) is only supported for versions

5.6 and later.

• PostgreSQL 9.4 and later. Note that Change Data Capture (CDC) is only supported for versions 9.4.9

and higher and 9.5.4 and higher. The rds.logical_replication

parameter, which is required for CDC, is supported only in these versions and later.

• MariaDB (supported as a MySQL-compatible data source)

• Amazon Aurora (supported as a MySQL-compatible data source)

Topics

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Using Oracle as a Source

•

Using an Oracle Database as a Source for AWS Database Migration Service (p. 68)

•

Using a Microsoft SQL Server Database as a Source for AWS Database Migration Service (p. 76)

•

Using a PostgreSQL Database as a Source for AWS Database Migration Service (p. 80)

•

Using a MySQL-Compatible Database as a Source for AWS Database Migration Service (p. 85)

•

Using a SAP ASE Database as a Source for AWS Database Migration Service (p. 88)

•

Using MongoDB as a Source for AWS Database Migration Service (p. 90)

Using an Oracle Database as a Source for AWS

Database Migration Service

You can migrate data from one or many Oracle databases using AWS Database Migration Service (AWS

DMS). With an Oracle database as a source, you can migrate data to either another Oracle database or one of the other supported databases.

AWS DMS supports as a migration source all Oracle database editions for versions 10.2 and later, 11g, and up to 12.1 on an on-premises or EC2 instance, and all Oracle database editions for versions 11g

(versions 11.2.0.3.v1 and later) and up to 12.1 for an Amazon RDS DB instance. An Oracle account with the speciﬁc access privileges is required.

You can use SSL to encrypt connections between your Oracle endpoint and the replication instance. For more information on using SSL with an Oracle endpoint, see

Using SSL With AWS Database Migration

Service (p. 56)

If you plan to use change data capture (CDC) from your Oracle database, you need to set up supplemental logging. For information on setting up supplemental logging, see

Conﬁguring Oracle as a

Source (p. 73) .

For additional details on working with Oracle databases and AWS DMS, see the following sections.

Topics

•

Using Oracle LogMiner or Oracle Binary Reader for Change Data Capture (CDC) (p. 68)

•

Limitations on Using Oracle as a Source for AWS Database Migration Service (p. 71)

•

Supported Compression Methods (p. 71)

•

User Account Privileges Required for Using Oracle as a Source for AWS Database Migration

Service (p. 71)

•

Conﬁguring Oracle as a Source (p. 73)

•

Conﬁguring Oracle on an Amazon RDS DB Instance as a Source (p. 74)

Using Oracle LogMiner or Oracle Binary Reader for

Change Data Capture (CDC)

Oracle oﬀers two methods for reading the logs when doing change processing: Oracle LogMiner and

Oracle Binary Reader.

By default, AWS DMS uses Oracle LogMiner for change data capture (CDC). Alternatively, you can choose to use the Oracle Binary Reader. The Oracle Binary Reader bypasses LogMiner and reads the logs directly.

To enable the Binary Reader, you need to modify your source connection to include the following extra connection parameters:

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Using Oracle LogMiner or Oracle Binary

Reader for Change Data Capture (CDC) useLogminerReader=N; useBfile=Y

To enable Logminer, you need to modify your source connection to include the following extra connection parameter or leave the Extra Connection Attribute blank (Logminer is the default): useLogminerReader=Y

To use LogMiner or Binary Reader, you must set the correct permissions. For information on setting these permissions, see the following section,

Access Privileges Required for Change Data Capture (CDC) on an

Oracle Source Database (p. 69)

The advantages to using LogMiner with AWS DMS, instead of Binary Reader, include the following:

• LogMiner supports most Oracle options, such as encryption options and compression options. Binary

Reader doesn't support all Oracle options, in particular options for encryption and compression.

• LogMiner oﬀers a simpler conﬁguration, especially compared to Oracle Binary Reader's direct access setup or if the redo logs are on Automatic Storage Management (ASM).

• LogMiner can be used with Oracle sources that use Oracle transparent data encryption (TDE).

The advantages to using Binary Reader with AWS DMS, instead of LogMiner, include the following:

• For migrations with a high volume of changes, LogMiner might have some I/O or CPU impact on the computer hosting the Oracle source database. Binary Reader has less chance of having I/O or CPU impact.

• For migrations with a high volume of changes, CDC performance is usually much better when using

Binary Reader compared with using Oracle LogMiner.

• Binary Reader supports CDC for LOBS in Oracle version 12c. LogMiner does not.

In general, use Oracle LogMiner for migrating your Oracle database unless you have one of the following situations:

• You need to run several migration tasks on the source Oracle database.

• The volume of changes or the REDO log volume on the source Oracle database is large.

• You need to propagate changes to LOBs in Oracle 12c.

Access Privileges Required for Change Data Capture (CDC) on an

Oracle Source Database

The access privileges required depend on whether you use Oracle LogMiner or Oracle Binary Reader for

CDC.

The following privileges must be granted to the user account used for data migration when using Oracle

LogMiner for change data capture (CDC) with an Oracle source database:

For Oracle versions prior to version 12c, grant the following:

• CREATE SESSION

• EXECUTE on DBMS_LOGMNR

• SELECT ANY TRANSACTION

• SELECT on V_$LOGMNR_LOGS

• SELECT on V_$LOGMNR_CONTENTS

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Using Oracle LogMiner or Oracle Binary

Reader for Change Data Capture (CDC)

For Oracle versions 12c and higher, grant the following:

• LOGMINING (for example, GRANT LOGMINING TO

<user account>

)

• CREATE SESSION

• EXECUTE on DBMS_LOGMNR

• SELECT on V_$LOGMNR_LOGS

• SELECT on V_$LOGMNR_CONTENTS

The following privileges must be granted to the user account used for data migration when using Oracle

Binary Reader for change data capture (CDC) with an Oracle source database:

• SELECT on v_$transportable_platform

Grant the SELECT on v_$transportable_platform privilege if the Redo logs are stored in Automatic

Storage Management (ASM) and accessed by AWS DMS from ASM.

• BFILE read - Used when AWS DMS does not have ﬁle-level access to the Redo logs, and the Redo logs are not accessed from ASM.

• DBMS_FILE_TRANSFER package - Used to copy the Redo log ﬁles to a temporary folder (in which case the EXECUTE ON DBMS_FILE_TRANSFER privilege needs to be granted as well)

• DBMS_FILE_GROUP package - Used to delete the Redo log ﬁles from a temporary/alternate folder (in which case the EXECUTE ON DBMS_FILE_ GROUP privilege needs to be granted as well).

• CREATE ANY DIRECTORY

Oracle ﬁle features work together with Oracle directories. Each Oracle directory object includes the name of the folder containing the ﬁles which need to be processed.

If you want AWS DMS to create and manage the Oracle directories, you need to grant the CREATE ANY

DIRECTORY privilege specified above. Note that the directory names will be prefixed with amazon_. If you do not grant this privilege, you need to create the corresponding directories manually. If you create the directories manually and the Oracle user specified in the Oracle Source endpoint is not the user that created the Oracle Directories, grant the READ on DIRECTORY privilege as well.

If the Oracle source endpoint is conﬁgured to copy the Redo log ﬁles to a temporary folder, and the

Oracle user speciﬁed in the Oracle source endpoint is not the user that created the Oracle directories, the following additional privileges are required:

• READ on the Oracle directory object speciﬁed as the source directory

• WRITE on the directory object speciﬁed as the destination directory in the copy process.

Limitations for Change Data Capture (CDC) on an Oracle Source

Database

The following limitations apply when using an Oracle database as a source for AWS Database Migration

Service (AWS DMS) change data capture.

• Oracle LogMiner, which AWS DMS uses for change data capture (CDC), doesn't support updating large binary objects (LOBs) when the UPDATE statement updates only LOB columns.

• For Oracle 11, Oracle LogMiner doesn't support the UPDATE statement for XMLTYPE and LOB columns.

• On Oracle 12c, LogMiner does not support LOB columns.

• AWS DMS doesn't capture changes made by the Oracle DBMS_REDEFINITION package, such as changes to table metadata and the OBJECT_ID value.

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Limitations on Oracle as a Source

• AWS DMS doesn't support index-organized tables with an overﬂow segment in change data capture

(CDC) mode when using BFILE. An example is when you access the redo logs without using LogMiner.

• AWS DMS doesn't support table clusters when you use Oracle Binary Reader.

• AWS DMS doesn't support virtual columns.

Limitations on Using Oracle as a Source for AWS

Database Migration Service

The following limitations apply when using an Oracle database as a source for AWS Database Migration

Service (AWS DMS). If you are using Oracle LogMiner or Oracle Binary Reader for change data capture

(CDC), see

Limitations for Change Data Capture (CDC) on an Oracle Source Database (p. 70) for

additional limitations.

• AWS DMS supports Oracle transparent data encryption (TDE) tablespace encryption and AWS Key

Management Service (AWS KMS) encryption when used with Oracle LogMiner. All other forms of encryption are not supported.

• AWS DMS supports the rename table <table name> to <new table name>

syntax with Oracle version

11 and higher.

• Oracle source databases columns created using explicit CHAR Semantics are transferred to a target

Oracle database using BYTE semantics. You must create tables containing columns of this type on the target Oracle database before migrating.

• AWS DMS doesn't replicate data changes resulting from partition or subpartition operations (ADD,

DROP, EXCHANGE, and TRUNCATE). To replicate such changes, you need to reload the table being replicated. AWS DMS replicates any future data changes to newly added partitions without your needing to reload the table again. However, UPDATE operations on old data records in these partitions fail and generate a

0 rows affected

warning.

• The data deﬁnition language (DDL) statement

ALTER TABLE ADD <column> <data_type> DEFAULT

doesn't replicate the default value to the target, and the new column in the target is set to NULL.

If the new column is nullable, Oracle updates all the table rows before logging the DDL itself. As a result, AWS DMS captures the changes to the counters but doesn't update the target. Because the new column is set to NULL, if the target table has no primary key or unique index, subsequent updates generate a

0 rows affected

warning.

• Data changes resulting from the

CREATE TABLE AS

statement are not supported. However, the new table is created on the target.

• When limited-size LOB mode is enabled, AWS DMS replicates empty LOBs on the Oracle source as

NULL values in the target.

• When AWS DMS begins CDC, it maps a timestamp to the Oracle system change number (SCN). By default, Oracle keeps only five days of the timestamp to SCN mapping. Oracle generates an error if the timestamp specified is too old (greater than the five day retention). For more information, see the

Oracle documentation .

Supported Compression Methods

AWS Database Migration Service supports all compression methods supported by LogMiner.

User Account Privileges Required for Using Oracle as a Source for AWS Database Migration Service

To use an Oracle database as a source in an AWS DMS task, the user speciﬁed in the AWS

DMS Oracle database deﬁnitions must be granted the following privileges in the Oracle

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

User Account Privileges for Oracle as a Source database. To grant privileges on Oracle databases on Amazon RDS, use the stored procedure rdsadmin.rdsadmin_util.grant_sys_object

. For more information, see Granting SELECT or EXECUTE privileges to SYS Objects .

Note

When granting privileges, use the actual name of objects (for example,

V_$OBJECT

including the underscore), not the synonym for the object (for example,

V$OBJECT

without the underscore).

• SELECT ANY TRANSACTION

• SELECT on V_$ARCHIVED_LOG

• SELECT on V_$LOG

• SELECT on V_$LOGFILE

• SELECT on V_$DATABASE

• SELECT on V_$THREAD

• SELECT on V_$PARAMETER

• SELECT on V_$NLS_PARAMETERS

• SELECT on V_$TIMEZONE_NAMES

• SELECT on V_$TRANSACTION

• SELECT on ALL_INDEXES

• SELECT on ALL_OBJECTS

• SELECT on DBA_OBJECTS (required if the Oracle version is earlier than 11.2.0.3)

• SELECT on ALL_TABLES

• SELECT on ALL_USERS

• SELECT on ALL_CATALOG

• SELECT on ALL_CONSTRAINTS

• SELECT on ALL_CONS_COLUMNS

• SELECT on ALL_TAB_COLS

• SELECT on ALL_IND_COLUMNS

• SELECT on ALL_LOG_GROUPS

• SELECT on SYS.DBA_REGISTRY

• SELECT on SYS.OBJ$

• SELECT on DBA_TABLESPACES

• SELECT on ALL_TAB_PARTITIONS

• SELECT on ALL_ENCRYPTED_COLUMNS

For the requirements speciﬁed following, grant the additional privileges named:

• If views are exposed, grant SELECT on ALL_VIEWS.

• When you use a speciﬁc table list, for each replicated table grant SELECT.

• When you use a pattern for a speciﬁc table list, grant SELECT ANY TABLE.

• When you add supplemental logging, grant ALTER ANY TABLE.

• When you add supplemental logging and you use a speciﬁc table list, grant ALTER for each replicated table.

• When migrating from Oracle RAC, grant permission to materialized views with the preﬁxes g_$ and v_

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Conﬁguring Oracle as a Source

Before using an Oracle database as a data migration source, you need to perform several tasks. For an

Oracle database on Amazon RDS, see the following section. You can also use extra connection attributes to conﬁgure the Oracle source. For more information about extra connection attributes, see

Using Extra

Connection Attributes with AWS Database Migration Service (p. 197)

For an Oracle database on premises or on an Amazon EC2 instance, you should do the following:

• Provide Oracle Account Access – You must provide an Oracle user account for AWS Database

Migration Service. The user account must have read/write privileges on the Oracle database, as speciﬁed in

User Account Privileges Required for Using Oracle as a Source for AWS Database Migration

Service (p. 71)

• Ensure that ARCHIVELOG Mode Is On – Oracle can run in two diﬀerent modes, the ARCHIVELOG mode and the NOARCHIVELOG mode. To use Oracle with AWS Database Migration Service, the database in question must be in ARCHIVELOG mode.

• Set Up Supplemental Logging – The following steps, required only when using change data capture

(CDC), show how to set up supplemental logging for an Oracle database. For information on setting

up supplemental logging on a database on an Amazon RDS DB instance, see Conﬁguring Oracle on an

Amazon RDS DB Instance as a Source (p. 74) .

To set up supplemental logging for an Oracle database

1. Determine if supplemental logging is enabled for the database:

• Run the following query:

SELECT name, value, description FROM v$parameter WHERE name = 'compatible';

The return result should be from

GE to 9.0.0

• Run the following query:

SELECT supplemental_log_data_min FROM v$database;

The returned result should be

YES

IMPLICIT

• Enable supplemental logging by running the following query:

ALTER DATABASE ADD SUPPLEMENTAL LOG DATA

2. Make sure that the required supplemental logging is added for each table:

• If a primary key exists, supplemental logging must be added for the primary key, either by using the format to add supplemental logging on the primary key or by adding supplemental logging on the primary key columns.

• If no primary key exists and the table has a single unique index, then all of the unique index’s columns must be added to the supplemental log. Using

SUPPLEMENTAL LOG DATA (UNIQUE

INDEX) COLUMNS

doesn't add the unique index columns to the log.

• If no primary key exists and the table has multiple unique indexes, AWS DMS selects the ﬁrst unique index. AWS DMS uses the ﬁrst index in an alphabetically ordered ascending list in this case. Supplemental logging must be added on the selected index's columns. Using

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Conﬁguring an RDS Oracle DB Instance as a Source

SUPPLEMENTAL LOG DATA (UNIQUE INDEX) COLUMNS doesn't add the unique index columns to the log.

• If there is no primary key and no unique index, supplemental logging must be added on all columns.

When the target table primary key or unique index is diﬀerent than the source table primary key or unique index, you should add supplemental logging manually on the source table columns that make up the target table primary key or unique index.

• If you change the target table primary key, you should add supplemental logging on the selected index's columns, instead of the columns of the original primary key or unique index.

3. Perform additional logging if necessary, for example if a ﬁlter is deﬁned for a table.

If a table has a unique index or a primary key, you need to add supplemental logging on each column that is involved in a ﬁlter if those columns are diﬀerent than the primary key or unique index columns. However, if ALL COLUMNS supplemental logging has been added to the table, you don't need to add any additional logging.

ALTER TABLE EXAMPLE.TABLE ADD SUPPLEMENTAL LOG GROUP example_log_group (ID,NAME)

ALWAYS;

Note

You can also turn on supplemental logging using a connection attribute. If you use this option, you still need to enable supplemental logging at the database level using the following statement:

ALTER DATABASE ADD SUPPLEMENTAL LOG DATA

For more information on setting a connection attribute, see

Oracle (p. 200)

Conﬁguring Oracle on an Amazon RDS DB Instance as a Source

Using an Oracle database on an Amazon RDS DB instance as a data migration source requires several settings on the DB instance, including the following:

• Set Up Supplemental Logging – AWS DMS requires database-level supplemental logging to be enabled. To enable database-level supplemental logging, run the following command: exec rdsadmin.rdsadmin_util.alter_supplemental_logging('ADD');

Note

You can also turn on supplemental logging using a connection attribute. If you use this option, you still need to enable supplemental logging at the database level using the following statement:

exec rdsadmin.rdsadmin_util.alter_supplemental_logging('ADD');

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Conﬁguring an RDS Oracle DB Instance as a Source

For more information on setting a connection attribute, see Oracle (p. 200)

In addition to running this command, we recommend that you turn on PRIMARY KEY logging at the database level to enable change capture for tables that have primary keys. To turn on PRIMARY KEY logging, run the following command: exec rdsadmin.rdsadmin_util.alter_supplemental_logging('ADD','PRIMARY KEY');

If you want to capture changes for tables that don't have primary keys, you should alter the table to add supplemental logging using the following command: alter table

table_name

add supplemental log data (ALL) columns;

Additionally, when you create new tables without specifying a primary key, you should either include a supplemental logging clause in the create statement or alter the table to add supplemental logging.

The following command creates a table and adds supplemental logging: create table

table_name

(

column data type

, supplemental log data(ALL) columns);

If you create a table and later add a primary key, you need to add supplemental logging to the table.

The following command alters the table to include supplemental logging: alter table

table_name

add supplemental log data (PRIMARY KEY) columns;

• Enable Automatic Backups – For information on setting up automatic backups, see the

Amazon RDS

User Guide

• Set Up Archiving – To retain archived redo logs of your Oracle database instance, which lets AWS DMS retrieve the log information using LogMiner, execute the following command. In this example, logs are kept for 24 hours.

exec rdsadmin.rdsadmin_util.set_configuration('archivelog retention hours',24);

Make sure that your storage has suﬃcient space for the archived redo logs during the speciﬁed period.

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Using SQL Server as a Source

Using a Microsoft SQL Server Database as a Source for AWS Database Migration Service

You can migrate data from one or many Microsoft SQL Server databases using AWS Database Migration

Service (AWS DMS). With a SQL Server database as a source, you can migrate data to either another SQL

Server database or one of the other supported databases.

AWS DMS supports, as a source, on-premises and Amazon EC2 instance databases for Microsoft SQL

Server versions 2005, 2008, 2008R2, 2012, and 2014. The Enterprise, Standard, Workgroup, and

Developer editions are supported. The Web and Express editions are not supported.

AWS DMS supports, as a source, Amazon RDS DB instance databases for SQL Server versions 2008R2,

2012, and 2014. The Enterprise and Standard editions are supported. Change data capture (CDC) is not supported for source databases on Amazon RDS. The Web, Workgroup, Developer, and Express editions are not supported.

You can have the source SQL Server database installed on any computer in your network. A SQL Server account with the appropriate access privileges to the source database for the type of task you chose is also required for use with AWS DMS.

You can use SSL to encrypt connections between your SQL Server endpoint and the replication instance.

For more information on using SSL with a SQL Server endpoint, see Using SSL With AWS Database

Migration Service (p. 56)

To capture changes from a source SQL Server database, the database must be conﬁgured for full backups and must be either the Enterprise, Developer, or Standard Edition. For a list of requirements

and limitations when using CDC with SQL Server, see Special Limitations When Capturing Data Changes

(CDC) from a SQL Server Source (p. 77) .

For additional details on working with SQL Server source databases and AWS DMS, see the following.

Topics

•

General Limitations on Using SQL Server as a Source for AWS Database Migration Service (p. 76)

•

Special Limitations When Capturing Data Changes (CDC) from a SQL Server Source (p. 77)

•

Supported Compression Methods (p. 78)

•

Working with Microsoft SQL Server AlwaysOn Availability Groups (p. 78)

•

Conﬁguring a Microsoft SQL Server Database as a Replication Source for AWS Database Migration

Service (p. 78)

•

Using MS-Replication to Capture Data Changes in Microsoft SQL Server (p. 79)

•

Using MS-CDC to Capture Data Changes in Microsoft SQL Server (p. 79)

•

Capturing Changes If You Can't Use MS-Replication or MS-CDC (p. 80)

General Limitations on Using SQL Server as a Source for AWS Database Migration Service

The following limitations apply when using a SQL Server database as a source for AWS DMS:

• The identity property for a column is not migrated to a target database column.

• Changes to rows with more than 8000 bytes of information, including header and mapping information, are not processed correctly due to limitations in the SQL Server TLOG buﬀer size.

• The Microsoft SQL Server endpoint does not support the use of sparse tables.

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

SQL Server CDC Limitations

• Windows Authentication is not supported.

• Changes to computed ﬁelds in a Microsoft SQL Server are not replicated.

• Microsoft SQL Server partition switching is not supported.

• A clustered index on the source will be created as a non-clustered index on target.

• When using the WRITETEXT and UPDATETEXT utilities, AWS DMS does not capture events applied on the source database.

• The following data manipulation language (DML) pattern is not supported:

SELECT <*> INTO <new_table> FROM <existing_table>

• When using Microsoft SQL Server as a source, column-level encryption is not supported.

• Due to a known issue with Microsoft SQL Server 2008 and Microsoft SQL Server 2008 R2, AWS

Database Migration Service doesn't support server level audits on Microsoft SQL Server 2008 and

Microsoft SQL Server 2008 R2 as a source endpoint.

For example, running the following command causes AWS Database Migration Service to fail:

USE [master]

ALTER SERVER AUDIT [my_audit_test-20140710] WITH (STATE=on)

Special Limitations When Capturing Data Changes

(CDC) from a SQL Server Source

The following limitations apply speciﬁcally when trying to capture changes from a SQL Server database as a source for AWS DMS:

• You must use either the Enterprise, Standard, or Developer Edition.

• SQL Server must be conﬁgured for full backups and a backup must be made before beginning to replicate data.

• Recovery Model must be set to Bulk logged or Full.

• Microsoft SQL Server backup to multiple disks is not supported. If the backup is defined to write the database backup to multiple files over different disks, AWS DMS will not be able to read the data and the AWS DMS task will fail.

• Microsoft SQL Server Replication Publisher deﬁnitions for the source database used in a DMS CDC task are not removed when you remove a task. A Microsoft SQL Server system administrator must delete these deﬁnitions from Microsoft SQL Server.

• During CDC, AWS DMS needs to look up SQL Server transaction log backups to read changes. AWS

DMS does not support using SQL Server transaction log backups that were created using third party backup software.

• SQL Server does not capture changes on newly created tables until they have been published. When tables are added to a SQL Server source, AWS DMS manages the creation of the publication. However, this process might take several minutes. Operations made to newly created tables during this delay will not be captured or replicated to the target.

• The AWS DMS user account must have the sysAdmin ﬁxed server role on the Microsoft SQL Server database you are connecting to.

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Supported Compression Methods

• AWS DMS change data capture requires FULLOGGING to be turned on in SQL Server; the only way to turn on FULLLOGGING in SQL Server is to either enable MS-REPLICATION or CHANGE DATA CAPTURE

(CDC).

• The SQL Server tlog cannot be reused until the changes have been processed.

• CDC operations are not supported on memory optimized tables. This limitation applies to Microsoft

SQL Server 2014 (when the feature was ﬁrst introduced) and above.

Supported Compression Methods

The following table shows the compression methods AWS DMS supports for each Microsoft SQL Server version.

Microsoft SQL Server Version

2005

2008

2012

2014

Row/Page Compression (at

Partition Level)

Yes

Vardecimal Storage Format

Note

Sparse columns and columnar structure compression are not supported.

Working with Microsoft SQL Server AlwaysOn

Availability Groups

The Microsoft SQL Server AlwaysOn Availability Groups feature is a high-availability and disasterrecovery solution that provides an enterprise-level alternative to database mirroring.

To use AlwaysOn Availability Groups as a source in AWS DMS, do the following:

• Enable the Distribution option on all Microsoft SQL Server instances in your Availability Replicas.

• In the AWS DMS console, open the Microsoft SQL Server source database settings. For Server Name, specify the Domain Name Service (DNS) name or IP address that was conﬁgured for the Availability

Group Listener.

When you start an AWS Database Migration Service task for the ﬁrst time, it might take longer than usual to start because the creation of the table articles is being duplicated by the Availability Groups

Server.

Conﬁguring a Microsoft SQL Server Database as a Replication Source for AWS Database Migration

Service

You can conﬁgure a Microsoft SQL Server database as a replication source for AWS Database Migration

Service (AWS DMS). For the most complete replication of changes, we recommend that you use the

Enterprise, Standard, or Developer edition of Microsoft SQL Server. One of these versions is required

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Using MS-Replication to Capture Data

Changes in Microsoft SQL Server because these are the only versions that include MS-Replication(EE,SE) and MS-CDC(EE,DEV). The source

SQL Server must also be conﬁgured for full backups. In addition, AWS DMS must connect with a user (a

SQL Server instance login) that has the sysAdmin ﬁxed server role on the SQL Server database you are connecting to.

Following, you can ﬁnd information about conﬁguring SQL Server as a replication source for AWS DMS.

Using MS-Replication to Capture Data Changes in

Microsoft SQL Server

To use MS-REPLICATION to replicate changes, each source table must have a primary key. If a source table doesn't have a primary key, you can use MS-CDC for capturing changes. If you haven't previously enabled MS-REPLICATION, you must enable your SQL Server database to use MS-REPLICATION. To enable MS-REPLICATION, take the steps following or see the Microsoft SQL Server documentation .

Setting up MS_REPLICATION adds a new SYSTEM database called

Distribution

to your source SQL

Server database.

To open the Conﬁgure Distribution wizard from Microsoft SQL Server

1. In Microsoft SQL Server Management Studio, open the context (right-click) menu for the

Replication

folder, and then choose Conﬁgure Distribution.

2. In the

Distributor step, choose <Microsoft SQL Server Name> will act as its own distributor. SQL

Server creates a distribution database and log.

Using MS-CDC to Capture Data Changes in Microsoft

SQL Server

If you need to replicate tables that don't have a primary key, the Use MS-CDC and Do Not Use MS-

Replication or MS-CDC options are available, as described following.

Important

Replicating tables that don't have a primary key or a unique index can adversely aﬀect performance, because additional database resources are required to capture the changes.

However, you can prevent performance issues related to the absence of primary keys or a unique index by manually adding indexes to the target tables.

Note

SQL Server might not delete log entries. Log entries are not reused unless they are replicated and backed up.

Setting Up MS-CDC

To set up MS-CDC, you ﬁrst need to enable MS-CDC for the database by running the following command.

use [DBname]

EXEC sys.sp_cdc_enable_db

Next, you need to enable MS-CDC for each of the source tables by running the following command.

EXECUTE sys.sp_cdc_enable_table @source_schema = N'MySchema', @source_name =

N'MyTable', @role_name = NULL;

For more information on setting up MS-CDC for speciﬁc tables, see the Microsoft SQL Server documentation .

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Capturing Changes If You Can't

Use MS-Replication or MS-CDC

Capturing Changes If You Can't Use MS-Replication or

MS-CDC

If your database is not set up for MS-REPLICATION or MS-CDC, and it is listed as a supported conﬁguration for CDC as discussed at the beginning of this section, you can still capture some changes to tables. However, such a setup only captures INSERT and DELETE data manipulation language (DML) events. UPDATE and TRUNCATE TABLE events are ignored. Depending on what operations occur, this setup can result in the target database no longer being consistent with the source.

Also in this setup, a DELETE event will not be applied to a row that has previously been modiﬁed on the source but not on the target (a row where a previous UPDATE event that was ignored).

Using a PostgreSQL Database as a Source for AWS

Database Migration Service

You can migrate data from one or many PostgreSQL databases using AWS Database Migration Service

(AWS DMS). With a PostgreSQL database as a source, you can migrate data to either another PostgreSQL database or one of the other supported databases. AWS DMS supports a PostgreSQL version 9.4

database as a source for on-premises databases, databases on an EC2 instance, and databases on an

Amazon RDS DB instance.

You can use SSL to encrypt connections between your PostgreSQL endpoint and the replication instance.

For more information on using SSL with a PostgreSQL endpoint, see Using SSL With AWS Database

Migration Service (p. 56)

AWS DMS supports change data capture (CDC) for PostgreSQL tables with primary keys; if a table does not have a primary key, the WAL logs do not include a before image of the database row and AWS DMS cannot update the table.

AWS DMS supports CDC on Amazon RDS PostgreSQL databases when the DB instance is conﬁgured to use logical replication. Amazon RDS supports logical replication for a PostgreSQL DB instance version

9.4.9 and higher and 9.5.4 and higher.

For additional details on working with PostgreSQL databases and AWS DMS, see the following sections.

Topics

•

Prerequisites for Using a PostgreSQL Database as a Source for AWS Database Migration

Service (p. 80)

•

Security Requirements When Using a PostgreSQL Database as a Source for AWS Database Migration

Service (p. 81)

•

Limitations on Using a PostgreSQL Database as a Source for AWS Database Migration

Service (p. 81)

•

Setting Up an Amazon RDS PostgreSQL DB Instance as a Source (p. 82)

•

Removing AWS Database Migration Service Artifacts from a PostgreSQL Source Database (p. 84)

•

Additional Conﬁguration Settings When Using a PostgreSQL Database as a Source for AWS Database

Migration Service (p. 84)

Prerequisites for Using a PostgreSQL Database as a

Source for AWS Database Migration Service

For a PostgreSQL database to be a source for AWS DMS, you should do the following:

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Security Requirements for PostgreSQL as a Source

• Use a PostgreSQL database that is version 9.4.x or later.

• Grant superuser permissions for the user account speciﬁed for the PostgreSQL source database.

• Add the IP address of the AWS DMS replication server to the pg_hba.conf

conﬁguration ﬁle.

• Set the following parameters and values in the postgresql.conf

conﬁguration ﬁle:

• Set wal_level = logical

• Set max_replication_slots >=1

The max_replication_slots

value should be set according to the number of tasks that you want to run. For example, to run ﬁve tasks you need to set a minimum of ﬁve slots. Slots open automatically as soon as a task starts and remain open even when the task is no longer running. You need to manually delete open slots.

• Set max_wal_senders >=1

The max_wal_senders

parameter sets the number of concurrent tasks that can run.

• Set wal_sender_timeout =0

The wal_sender_timeout

parameter terminates replication connections that are inactive longer than the speciﬁed number of milliseconds. Although the default is 60 seconds, we recommend that you set this parameter to zero, which disables the timeout mechanism.

Security Requirements When Using a PostgreSQL

Database as a Source for AWS Database Migration

Service

The only security requirement when using PostgreSQL as a source is that the user account speciﬁed must be a registered user in the PostgreSQL database.

Limitations on Using a PostgreSQL Database as a

Source for AWS Database Migration Service

The following change data capture (CDC) limitations apply when using PostgreSQL as a source for AWS

DMS:

• A captured table must have a primary key. If a table doesn't have a primary key, AWS DMS ignores

DELETE and UPDATE record operations for that table.

• AWS DMS ignores an attempt to update a primary key segment. In these cases, the target identiﬁes the update as one that didn't update any rows, but since the results of updating a primary key in

PostgreSQL is unpredictable, no records are written to the exceptions table.

• AWS DMS doesn't support the Start Process Changes from Timestamp run option.

• AWS DMS supports full load and change processing on Amazon RDS for PostgreSQL. For information

on how to prepare a PostgreSQL DB instance and to set it up for using CDC, see Setting Up an Amazon

RDS PostgreSQL DB Instance as a Source (p. 82) .

• AWS DMS doesn't map some PostgreSQL data types, including the JSON data type. The JSON is converted to CLOB.

• Replication of multiple tables with the same name but where each name has a diﬀerent case (for example table1, TABLE1, and Table1) can cause unpredictable behavior, and therefore AWS DMS doesn't support it.

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Setting Up an Amazon RDS

PostgreSQL DB Instance as a Source

• AWS DMS supports change processing of CREATE, ALTER, and DROP DDL statements for tables unless the tables are held in an inner function or procedure body block or in other nested constructs.

For example, the following change is not captured:

CREATE OR REPLACE FUNCTION attu.create_distributors1() RETURNS void

LANGUAGE plpgsql

AS $$

BEGIN create table attu.distributors1(did serial PRIMARY KEY,name varchar(40) NOT NULL);

END;

$$;

• AWS DMS doesn't support change processing of TRUNCATE operations.

• AWS DMS doesn't support replication of partitioned tables. When a partitioned table is detected, the following occurs:

• The endpoint reports a list of parent and child tables.

• AWS DMS creates the table on the target as a regular table with the same properties as the selected tables.

• If the parent table in the source database has the same primary key value as its child tables, a

“duplicate key” error is generated.

Note

To replicate partitioned tables from a PostgreSQL source to a PostgreSQL target, you first need to manually create the parent and child tables on the target. Then you define a separate task to replicate to those tables. In such a case, you set the task configuration to

Truncate before

loading.

Setting Up an Amazon RDS PostgreSQL DB Instance as a Source

You can use the AWS master user account for the PostgreSQL DB instance as the user account for the

PostgreSQL source endpoint for AWS DMS . The master user account has the required roles that allow it to set up change data capture (CDC). If you use an account other than the master user account, the account must have the rds_superuser role and the rds_replication role. The rds_replication role grants permissions to manage logical slots and to stream data using logical slots.

If you don't use the master user account for the DB instance, you must create several objects from the master user account for the account that you use. For information about creating the needed objects, see

Migrating an Amazon RDS for PostgreSQL Database Without Using the Master User Account (p. 83)

Using CDC with an Amazon RDS for PostgreSQL DB Instance

You can use PostgreSQL's native logical replication feature to enable CDC during a database migration of an Amazon RDS PostgreSQL DB instance. This approach reduces downtime and ensures that the target database is in sync with the source PostgreSQL database. Amazon RDS supports logical replication for a

PostgreSQL DB instance version 9.4.9 and higher and 9.5.4 and higher.

To enable logical replication for an RDS PostgreSQL DB instance, do the following:

• In general, use the AWS master user account for the PostgreSQL DB instance as the user account for the PostgreSQL source endpoint. The master user account has the required roles that allow the it to set up CDC. If you use an account other than the master user account, you must create several objects

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Setting Up an Amazon RDS

PostgreSQL DB Instance as a Source

from the master account for the account that you use. For more information, see Migrating an Amazon

RDS for PostgreSQL Database Without Using the Master User Account (p. 83)

• Set the rds.logical_replication

parameter in your DB parameter group to 1. This is a static parameter that requires a reboot of the DB instance for the parameter to take eﬀect. As part of applying this parameter, AWS DMS sets the wal_level

, max_wal_senders

, max_replication_slots

, and max_connections

parameters. Note that these parameter changes can increase WAL generation so you should only set the rds.logical_replication

parameter when you are using logical slots.

Migrating an Amazon RDS for PostgreSQL Database Without

Using the Master User Account

If you don't use the master user account for the Amazon RDS PostgreSQL DB instance that you are using as a source, you need to create several objects to capture data deﬁnition language (DDL) events. You create these objects in the account other than the master account and then create a trigger in the master user account.

Note

If you set the captureDDL

parameter to N on the source endpoint, you will not have to create the following table/trigger on the source database.

Use the following procedure to create these objects. The user account other than the master account is referred to as the NoPriv account in this procedure.

1. Choose a schema where the objects will be created. The default schema is public

. Ensure that the schema exists and is accessible by the NoPriv account.

2. Log in to the PostgreSQL DB instance using the NoPriv account.

3. Create the table awsdms_ddl_audit

by running the following command, replacing <objects_schema> in the code following with the name of the schema to use: create table <objects_schema>.awsdms_ddl_audit

(

c_key bigserial primary key,

c_time timestamp, -- Informational

c_user varchar(64), -- Informational: current_user

c_txn varchar(16), -- Informational: current transaction

c_tag varchar(24), -- Either 'CREATE TABLE' or 'ALTER TABLE' or 'DROP TABLE'

c_oid integer, -- For future use - TG_OBJECTID

c_name varchar(64), -- For future use - TG_OBJECTNAME

c_schema varchar(64), -- For future use - TG_SCHEMANAME. For now - holds

current_schema

c_ddlqry text -- The DDL query associated with the current DDL event

)

4. Create the function awsdms_intercept_ddl

by running the following command, replacing

<objects_schema> in the code following with the name of the schema to use:

CREATE OR REPLACE FUNCTION <objects_schema>.awsdms_intercept_ddl()

RETURNS event_trigger

LANGUAGE plpgsql

AS $$

declare _qry text;

BEGIN

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Removing Artifacts from a PostgreSQL Source

if (tg_tag='CREATE TABLE' or tg_tag='ALTER TABLE' or tg_tag='DROP TABLE') then

SELECT current_query() into _qry;

insert into <objects_schema>.awsdms_ddl_audit

values

(

default,current_timestamp,current_user,cast(TXID_CURRENT()as

varchar(16)),tg_tag,0,'',current_schema,_qry

);

delete from <objects_schema>.awsdms_ddl_audit; end if;

END;

$$;

5. Log out of the NoPriv account and log in with an account that has the rds_superuser role assigned to it.

6. Create the event trigger awsdms_intercept_ddl

by running the following command:

CREATE EVENT TRIGGER awsdms_intercept_ddl ON ddl_command_end

EXECUTE PROCEDURE <objects_schema>.awsdms_intercept_ddl();

When you have completed the procedure preceding, you can create the AWS DMS source endpoint using the NoPriv account.

Removing AWS Database Migration Service Artifacts from a PostgreSQL Source Database

To capture DDL events, AWS DMS creates various artifacts in the PostgreSQL database when a migration task starts. When the task completes, you might want to remove these artifacts. To remove the artifacts, issue the following statements (in the order they appear), where

{AmazonRDSMigration}

is the schema in which the artifacts were created: drop event trigger awsdms_intercept_ddl;

Note that the event trigger doesn't belong to a speciﬁc schema.

drop function {AmazonRDSMigration}.awsdms_intercept_ddl() drop table {AmazonRDSMigration}.awsdms_ddl_audit

drop schema {AmazonRDSMigration}

Note

Dropping a schema should be done with extreme caution, if at all. Never drop an operational schema, especially not a public one.

Additional Conﬁguration Settings When Using a

PostgreSQL Database as a Source for AWS Database

Migration Service

You can add additional conﬁguration settings when migrating data from a PostgreSQL database in two way.

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Using MySQL as a Source

• You can add values to the Extra Connection attribute to capture DDL events and to specify the schema in which the operational DDL database artifacts are created. For more information, see

PostgreSQL (p. 199) .

• You can override connection string parameters. Select this option if you need to do either of the following:

• Specify internal AWS DMS parameters. Such parameters are rarely required and are therefore not exposed in the user interface.

• Specify pass-through (passthru) values for the speciﬁc database client. AWS DMS includes passthrough parameters in the connection sting passed to the database client.

Using a MySQL-Compatible Database as a Source for AWS Database Migration Service

You can migrate data from one or many MySQL, MariaDB, or Amazon Aurora databases using AWS

Database Migration Service. With a MySQL-compatible database as a source, you can migrate data to either another MySQL-compatible database or one of the other supported databases. MySQL versions

5.5, 5.6, and 5.7, as well as MariaDB and Amazon Aurora, are supported for on-premises, Amazon RDS, and Amazon EC2 instance databases. To enable change data capture (CDC) with Amazon RDS MySQL, you must use Amazon RDS MySQL version 5.6 or higher.

Note

Regardless of the source storage engine (MyISAM, MEMORY, etc.), AWS DMS creates the MySQLcompatible target table as an InnoDB table by default. If you need to have a table that uses a storage engine other than InnoDB, you can manually create the table on the MySQL-compatible target and migrate the table using the "Do Nothing" mode. For more information about the "Do

Nothing" mode, see

Full Load Task Settings (p. 123)

You can use SSL to encrypt connections between your MySQL-compatible endpoint and the replication

instance. For more information on using SSL with a MySQL-compatible endpoint, see Using SSL With

AWS Database Migration Service (p. 56) .

For additional details on working with MySQL-compatible databases and AWS Database Migration

Service, see the following sections.

Topics

•

Prerequisites for Using a MySQL Database as a Source for AWS Database Migration Service (p. 85)

•

Prerequisites for Using an Amazon RDS for MySQL Database as a Source for AWS Database Migration

Service (p. 87)

•

Limitations on Using a MySQL Database as a Source for AWS Database Migration Service (p. 87)

•

Security Requirements for Using a MySQL Database as a Source for AWS Database Migration

Service (p. 87)

•

Conﬁguring a MySQL Database as a Source for AWS Database Migration Service (p. 88)

Prerequisites for Using a MySQL Database as a Source for AWS Database Migration Service

Before you begin to work with a MySQL database as a source for AWS DMS, make sure that you have the following prerequisites:

• A MySQL account with the required security settings. For more information, see

Security Requirements for Using a MySQL Database as a Source for AWS Database Migration Service (p. 87)

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Prerequisites for Using MySQL as a Source

• A MySQL-compatible database with the tables that you want to replicate accessible in your network.

• MySQL Community Edition

• MySQL Standard Edition

• MySQL Enterprise Edition

• MySQL Cluster Carrier Grade Edition

• MariaDB

• Amazon Aurora

• If your source is an Amazon RDS MySQL or MariaDB DB instance or an Amazon Aurora cluster, you must enable automatic backups. For more information on setting up automatic backups, see the

Amazon RDS User Guide

• If you use change data capture (CDC), you must enable and conﬁgure binary logging. To enable binary logging, the following parameters must be conﬁgured in MySQL’s my.ini

(Windows) or my.cnf

(UNIX) ﬁle:

Parameter

server_id log-bin binlog_format expire_logs_days

Value

Set this parameter to a value of 1 or greater.

Set the path to the binary log ﬁle, such as log-bin=E:\MySql_Logs\BinLog

Don't include the ﬁle extension.

Set this parameter to row

Set this parameter to a value of 1 or greater. To prevent overuse of disk space, we recommend that you don't use the default value of 0.

binlog_checksum binlog_row_image

Set this parameter to none

Set this parameter to full

• In order to use change data capture (CDC) with an Amazon RDS MySQL DB instance as a source, AWS

DMS needs access to the binary logs. Amazon RDS is fairly aggressive in clearing binary logs from the

DB instance. To use CDC with a MySQL DB instance on Amazon RDS, you should increase the amount of time the binary logs remain on the MySQL DB instance. For example, to increase log retention to 24 hours, you would run the following command:

call mysql.rds_set_configuration('binlog retention hours', 24);

• To replicate clustered (NDB) tables using AWS Database Migration Service, the following parameters must be conﬁgured in MySQL’s my.ini

(Windows) or my.cnf

(UNIX) ﬁle. Replicating clustered (NDB) tables is required only when you use CDC.

Parameter Value

ndb_log_bin ndb_log_updated_only

Set this parameter to on

. This value ensures that changes in clustered tables are logged to the binary log.

ndb_log_update_as_write

Set this parameter to

OFF

. This value prevents writing UPDATE statements as INSERT statements in the binary log.

Set this parameter to

OFF

. This value ensures that the binary log contains the entire row and not just the changed columns.

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Prerequisites for Using Amazon RDS for MySQL as a Source

Prerequisites for Using an Amazon RDS for MySQL

Database as a Source for AWS Database Migration

Service

When using an Amazon RDS for MySQL database as a source for AWS DMS, make sure that you have the following prerequisites:

• You must enable automatic backups. For more information on setting up automatic backups, see the

Amazon RDS User Guide

• In order to use change data capture (CDC) with an Amazon RDS MySQL DB instance as a source, AWS

DMS needs access to the binary logs. Amazon RDS is fairly aggressive in clearing binary logs from the

DB instance. To use CDC with a MySQL DB instance on Amazon RDS, you should increase the amount of time the binary logs remain on the MySQL DB instance to 24 hours or greater. For example, to increase log retention to 24 hours, you would run the following command:

call mysql.rds_set_configuration('binlog retention hours', 24);

• The binlog_format parameter should be set to "ROW."

Limitations on Using a MySQL Database as a Source for AWS Database Migration Service

When using a MySQL database as a source, AWS DMS doesn't support the following:

• The DDL statements Truncate Partition, Drop Table, and Rename Table.

• Using an

ALTER TABLE

<table_name>

ADD COLUMN

<column_name>

statement to add columns to the beginning or the middle of a table.

• Capturing changes from tables whose names contain both uppercase and lowercase characters.

• The AR_H_USER header column.

• The AUTO_INCREMENT attribute on a column is not migrated to a target database column.

• Capturing changes when the binary logs are not stored on standard block storage. For example, CDC does not work when the binary logs are stored on Amazon S3.

Security Requirements for Using a MySQL Database as a Source for AWS Database Migration Service

As a security requirement, the AWS DMS user must have the ReplicationAdmin role with the following privileges:

• REPLICATION CLIENT – This privilege is required for change data capture (CDC) tasks only. In other words, full-load-only tasks don't require this privilege.

• REPLICATION SLAVE – This privilege is required for change data capture (CDC) tasks only. In other words, full-load-only tasks don't require this privilege.

•

SUPER – This privilege is required only in MySQL versions prior to 5.6.6.

The AWS DMS user must also have SELECT privileges for the source tables designated for replication.

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Conﬁguring a MySQL Database as a Source for AWS Database Migration Service

Conﬁguring a MySQL Database as a Source for AWS

Database Migration Service

You can use extra connection attributes to conﬁgure the MySQL source. For more information about

MySQL extra connection attributes, see

Using Extra Connection Attributes with AWS Database Migration

Service (p. 197)

Using a SAP ASE Database as a Source for AWS

Database Migration Service

You can migrate data from a SAP Adaptive Server Enterprise (ASE) database–formerly known as Sybase– using AWS Database Migration Service. With a SAP ASE database as a source, you can migrate data to any of the other supported AWS DMS target databases.

For additional details on working with SAP ASE databases and AWS Database Migration Service, see the following sections.

Topics

•

Prerequisites for Using a SAP ASE Database as a Source for AWS Database Migration

Service (p. 88)

•

Limitations on Using SAP ASE as a Source for AWS Database Migration Service (p. 88)

•

User Account Permissions Required for Using SAP ASE as a Source for AWS Database Migration

Service (p. 89)

•

Removing the Truncation Point (p. 89)

Prerequisites for Using a SAP ASE Database as a

Source for AWS Database Migration Service

For a SAP ASE database to be a source for AWS DMS, you should do the following:

• SAP ASE replication must be enabled for tables by using the sp_setreptable

command.

•

RepAgent

must be disabled on the SAP ASE database.

• When replicating to SAP ASE version 15.7 installed on a Windows EC2 instance conﬁgured with a non-

Latin language (for example, Chinese), AWS DMS requires SAP ASE 15.7 SP121 to be installed on the target SAP ASE machine.

Limitations on Using SAP ASE as a Source for AWS

Database Migration Service

The following limitations apply when using an SAP ASE database as a source for AWS Database

Migration Service (AWS DMS):

• Only one AWS DMS task can be run per SAP ASE database.

• Rename table is not supported, for example: sp_rename 'Sales.SalesRegion', 'SalesReg;

• Rename column is not supported, for example: sp_rename 'Sales.Sales.Region', 'RegID',

'COLUMN';

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

User Account Permissions for SAP ASE as a Source

• Zero values located at the end of binary data type strings are truncated when replicated to the target database. For example,

0x0000000000000000000000000100000100000000

in the source table will become

0x00000000000000000000000001000001

in the target table.

• AWS DMS creates the target table with columns that do not allow NULL values, if the database default is not to allow NULL values. Consequently, if a Full Load or CDC replication task contains empty values, errors will occur.

You can prevent these errors from occurring by allowing nulls in the source database by using the following commands: sp_dboption

<database name>

, 'allow nulls by default', 'true' go use

<database name>

CHECKPOINT go

• The reorg rebuild

index command is not supported.

User Account Permissions Required for Using SAP

ASE as a Source for AWS Database Migration Service

To use an SAP ASE database as a source in an AWS DMS task, the user speciﬁed in the AWS DMS SAP ASE database deﬁnitions must be granted the following permissions in the SAP ASE database.

• sa_role

• replication_role

• sybase_ts_role

• If you have set the enableReplication

connection property to

, then your must also be granted the sp_setreptable

permission. For more information on connection properties see Using Extra

Connection Attributes with AWS Database Migration Service (p. 197)

Removing the Truncation Point

When a task starts, AWS DMS establishes a

$replication_truncation_point

entry in the syslogshold system view, indicating that a replication process is in progress. While AWS DMS is working, it advances the replication truncation point at regular intervals, according to the amount of data that has already been copied to the target.

Once the

$replication_truncation_point

entry has been established, the AWS DMS task must be kept running at all times to prevent the database log from becoming excessively large. If you want to stop the

AWS DMS task permanently, the replication truncation point must be removed by issuing the following command: dbcc settrunc('ltm','ignore')

After the truncation point has been removed, the AWS DMS task cannot be resumed. The log continues to be truncated automatically at the checkpoints (if automatic truncation is set).

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Using MongoDB as a Source

Using MongoDB as a Source for AWS Database

Migration Service

AWS DMS supports MongoDB versions 2.6.x and 3.x as a database source. A MongoDB database is a

JSON document database where there are multiple MongoDB collections made up of JSON documents.

In MongoDB, a collection is somewhat equivalent to a relational database table and a JSON document is somewhat equivalent to a row in that relational database table. Internally, a JSON document is stored as a binary JSON (BSON) ﬁle in a compressed format that includes a type for each ﬁeld in the document.

Each document has a unique ID.

AWS DMS supports two migration modes when using MongoDB as a source:

Document Mode

In document mode, the MongoDB document is migrated “as is,” meaning that its JSON data becomes a single column in a target table named “_doc”.

You can optionally set the extractDocID

parameter to true to create a second column named “_id” that will act as the primary key. You must set this parameter to true if you are going to use change data capture (CDC).

Document mode is the default setting when you use MongoDB as a source. To explicitly specify document mode, add nestingLevel=NONE

to the extra connection attribute on the MongoDB source endpoint.

Here is how AWS DMS manages documents and collections in document mode:

• When adding a new collection, the collection is replication as a CREATE TABLE.

• Renaming a collection is not supported.

Table Mode

In table mode, AWS DMS scans a specified number of MongoDB documents and creates a set of all the keys and their types. This set is then used to create the columns of the target table. In this mode, a MongoDB document is transformed into a table data row. Each top level field is transformed into a column. For each MongoDB document, AWS DMS adds each key and type to the target table’s column set. Nested values are flattened into a column containing dot-separated key names. For example, a JSON document consisting of {"a" : {"b" : {"c": 1}}} is migrated into a column named a.b.c.

You can specify how many documents are scanned by setting the docsToInvestigate

parameter.

The default value is 1000. You can enable table mode by adding nestingLevel=ONE

to the extra connection attributes of the MongoDB source endpoint.

Here is how AWS DMS manages documents and collections in table mode:

• When you add a document to an existing collection, the document (row) is replicated. If there are ﬁelds that do not exist in the collection, those ﬁelds are not replicated.

• When you update a document, the updated document is replicated. If there are ﬁelds that do not exist in the collection, those ﬁelds are not replicated.

• Deleting a document is fully supported.

• Adding a new collection will not result in a new table on the target when done during a CDC task.

• Renaming a collection is not supported.

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Prerequisites When Using MongoDB as a

Source for AWS Database Migration Service

Prerequisites When Using MongoDB as a Source for

AWS Database Migration Service

The user account used for the MongoDB endpoint needs to have access to the operations log of the replica set you create.

Prerequisites When Using CDC with MongoDB as a

Source for AWS Database Migration Service

To use change data capture (CDC) with a MongoDB source, you you'll need to do several things. First, you deploy the replica set to create the operations log. Next, you create the system user administrator.

Finally, you set the extractDocID parameter to true to extract the document ID that is used during CDC.

The MongoDB operations log (oplog) is a special capped collection that keeps a rolling record of all operations that modify the data stored in your databases. MongoDB applies database operations on the

primary and then records the operations on the primary's oplog. The secondary members then copy and apply these operations in an asynchronous process.

Deploying a Replica Set for Use with CDC

You need to deploy a replica set to use MongoDB as an AWS DMS source. When you deploy the replica set, you create the operations log that is used for CDC. Do the following steps to deploy the replica set.

For more information, see the MongoDB documentation .

To deploy a replica set

1. Using the command line, connect to mongo.

mongo localhost

2. Run in mongo shell.

rs.initiate() root

3. Verify the deployment.

rs.conf()

4. Set a diﬀerent database path. Run with --dbpath the-path.

Next, create the system user administrator role.

To create the root user

1. Create a user to be the root account, as shown in the following code. In this example, we call that user root

use admin db.createUser(

{

user: "root",

pwd: "rootpass",

roles: [ { role: "root", db: "admin" } ]

}

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Security Requirements When Using MongoDB as a Source for AWS Database Migration Service

)

2. Stop mongod.

3. Restart mongod using the following command: mongod --replSet "rs0" --auth

4. Test that the operations log is readable using the following commands: mongo localhost/admin -u root -p rootpass mongo --authenticationDatabase admin -u root -p rootpass

5. If you are unable to read from operations log, run the following command: rs.initiate();

The ﬁnal requirement to use CDC with MongoDB is to set the extractDocID

parameter. Set the extractDocID

parameter to true to create a second column named “_id” that will act as the primary key.

Security Requirements When Using MongoDB as a

Source for AWS Database Migration Service

AWS DMS supports two authentication methods for MongoDB. The two authentication methods are used to encrypt the password, so they are only used when the authType

parameter is set to password.

The MongoDB authentication methods are:

• MONOGODB-CR — the default when using MongoDB 2.x authentication.

• SCRAM-SHA-1 — the default when using MongoDB version 3.x authentication.

If an authentication method is not speciﬁed, AWS DMS uses the default method for the version of the

MongoDB source. The two authentication methods are used to encrypt the password, so they are only used when the authType

parameter is set to password.

Limitations When Using MongoDB as a Source for

AWS Database Migration Service

The following are limitations when using MongoDB as a source for AWS DMS:

• When the extractDocID

parameter is set to true, the ID string cannot exceed 200 characters.

• The MongoDB parameter

ObjectId

is a string with a limit of 200 characters. It includes the JSON structure of the id object: { "_id" : { "$oid" : "581730b9e85de3f180fd571e" } } and is used as the primary key in the target database.

• Collection names cannot include the dollar symbol ($).

Conﬁguration Properties When Using MongoDB as a

Source for AWS Database Migration Service

When you set up your MongoDB source endpoint, you can specify additional conﬁguration settings attributes. Attributes are speciﬁed by key-=value pairs and separated by semicolons. For example, the

API Version API Version 2016-01-01

AWS Database Migration Service User Guide

Conﬁguration Properties When Using MongoDB as a Source for AWS Database Migration Service following code speciﬁes that user name and password are not used for authentication, and that table mode is used.

The following table describes the conﬁguration properties available when using MongoDB databases as an AWS Database Migration Service source database.

Attribute

Name

authType

Valid Values

PASSWORD authMechanism DEFAULT

MONGODB_CR nestingLevel

SCRAM_SHA_1

NONE extractDocID

ONE true authSource false docsToInvestigateA positive integer greater than 0.

A valid MongoDB database name.

Default Value and Description

PASSWORD – When NO is selected, user name and password parameters are not used and can be empty.