Microsoft SQL Server User manual

Keep Nulls or Use Default Values During Bulk Import (SQL Server)

Keep Identity Values When Bulk Importing Data (SQL Server)

Format Files for Importing or Exporting Data (SQL Server))

● Create a Format File (SQL Server)

● Use a Format File to Bulk Import Data (SQL Server)

● Use a Format File to Skip a Table Column (SQL Server)

● Use a Format File to Skip a Data Field (SQL Server)

● Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

Examples of Bulk Import and Export of XML Documents (SQL Server)

More examples and information

INSERT (Transact-SQL)

SELECT Clause (Transact-SQL) bcp Utility

Prepare to Bulk Import Data (SQL Server)

BULK INSERT (Transact-SQL)

Bulk Import and Export of Data (SQL Server)

OPENROWSET (Transact-SQL)

Examples of Bulk Import and Export of XML Documents (SQL Server)

Import Bulk Data by Using BULK INSERT or

OPENROWSET(BULK...) (SQL Server)

3/24/2017 • 5 min to read • Edit Online

THIS TOPIC APPLIES TO:

SQL Server (starting with 2008) Azure SQL Database Azure SQL Data

Warehouse Parallel Data Warehouse

This topic provides an overview of how to use the Transact-SQL BULK INSERT statement and the INSERT...SELECT *

FROM OPENROWSET(BULK...) statement to bulk import data from a data file into a SQL Server table. This topic also describes security considerations for using BULK INSERT and OPENROWSET(BULK…), and using these methods to bulk import from a remote data source.

NOTE When you use BULK INSERT or OPENROWSET(BULK…), it is important to understand how SQL Server version handles impersonation. For more information, see "Security Considerations," later in this topic.

BULK INSERT statement

BULK INSERT loads data from a data file into a table. This functionality is similar to that provided by the in option of the bcp command; however, the data file is read by the SQL Server process. For a description of the BULK

INSERT syntax, see BULK INSERT (Transact-SQL) .

BULK INSERT examples

BULK INSERT (Transact-SQL)

Keep Identity Values When Bulk Importing Data (SQL Server)

Keep Nulls or Use Default Values During Bulk Import (SQL Server)

Use Character Format to Import or Export Data (SQL Server)

Use Native Format to Import or Export Data (SQL Server)

Use Unicode Character Format to Import or Export Data (SQL Server)

Use Unicode Native Format to Import or Export Data (SQL Server)

Use a Format File to Skip a Table Column (SQL Server)

Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

OPENROWSET(BULK…) Function

The OPENROWSET bulk rowset provider is accessed by calling the OPENROWSET function and specifying the

BULK option. The OPENROWSET(BULK…) function allows you to access remote data by connecting to a remote data source, such as a data file, through an OLE DB provider.

Applies to:

OPENROWSET

is not available in SQL Database.

To bulk import data, call OPENROWSET(BULK…) from a SELECT…FROM clause within an INSERT statement. The basic syntax for bulk importing data is:

INSERT ... SELECT * FROM OPENROWSET(BULK...)

When used in an INSERT statement, OPENROWSET(BULK...) supports table hints. In addition to the regular table hints, such as TABLOCK, the BULK clause can accept the following specialized table hints: IGNORE_CONSTRAINTS

(ignores only the CHECK constraints), IGNORE_TRIGGERS, KEEPDEFAULTS, and KEEPIDENTITY. For more information, see Table Hints (Transact-SQL) .

For information about additional uses of the BULK option, see OPENROWSET (Transact-SQL) .

INSERT...SELECT * FROM OPENROWSET(BULK...) statements examples:

Examples of Bulk Import and Export of XML Documents (SQL Server)

Keep Identity Values When Bulk Importing Data (SQL Server)

Keep Nulls or Use Default Values During Bulk Import (SQL Server)

Use Character Format to Import or Export Data (SQL Server)

Use a Format File to Skip a Table Column (SQL Server)

Use a Format File to Skip a Data Field (SQL Server)

Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

Security considerations

If a user uses a SQL Server login, the security profile of the SQL Server process account is used. A login using SQL

Server authentication cannot be authenticated outside of the Database Engine. Therefore, when a BULK INSERT command is initiated by a login using SQL Server authentication, the connection to the data is made using the security context of the SQL Server process account (the account used by the SQL Server Database Engine service).

To successfully read the source data you must grant the account used by the SQL Server Database Engine, access to the source data. In contrast, if a SQL Server user logs on by using Windows Authentication, the user can read only those files that can be accessed by the user account, regardless of the security profile of the SQL Server process.

For example, consider a user who logged in to an instance of SQL Server by using Windows Authentication. For the user to be able to use BULK INSERT or OPENROWSET to import data from a data file into a SQL Server table, the user account requires read access to the data file. With access to the data file, the user can import data from the file into a table even if the SQL Server process does not have permission to access the file. The user does not have to grant file-access permission to the SQL Server process.

SQL Server and Microsoft Windows can be configured to enable an instance of SQL Server to connect to another instance of SQL Server by forwarding the credentials of an authenticated Windows user. This arrangement is known as impersonation or delegation. Understanding how SQL Server version handle security for user impersonation is important when you use BULK INSERT or OPENROWSET. User impersonation allows the data file to reside on a different computer than either the SQL Server process or the user. For example, if a user on

Computer_A has access to a data file on Computer_B, and the delegation of credentials has been set appropriately, the user can connect to an instance of SQL Server that is running on Computer_C, access the data file on Computer_B, and bulk import data from that file into a table on Computer_C.

Bulk importing from a remote data file

To use BULK INSERT or INSERT...SELECT * FROM OPENROWSET(BULK...) to bulk import data from another computer, the data file must be shared between the two computers. To specify a shared data file, use its universal naming convention (UNC) name, which takes the general form, \\Servername\Sharename\Path\Filename.

Additionally, the account used to access the data file must have the permissions that are required for reading the file on the remote disk.

For example, the following

BULK INSERT

statement bulk imports data into the

SalesOrderDetail

table of the

AdventureWorks

database from a data file that is named newdata.txt

. This data file resides in a shared folder named

\dailyorders

on a network share directory named salesforce

on a system named computer2

BULK INSERT AdventureWorks2012.Sales.SalesOrderDetail

FROM '\\computer2\salesforce\dailyorders\neworders.txt';

NOTE This restriction does not apply to the bcp utility because the client reads the file independently of SQL

Server.

Prerequisites for Minimal Logging in Bulk Import

3/24/2017 • 2 min to read • Edit Online

THIS TOPIC APPLIES TO: SQL Server (starting with 2016) Azure SQL Database Azure SQL Data

Warehouse Parallel Data Warehouse

For a database under the full recovery model, all row-insert operations that are performed by bulk import are fully logged in the transaction log. Large data imports can cause the transaction log to fill rapidly if the full recovery model is used. In contrast, under the simple recovery model or bulk-logged recovery model, minimal logging of bulk-import operations reduces the possibility that a bulk-import operation will fill the log space. Minimal logging is also more efficient than full logging.

The bulk-logged recovery model is designed to temporarily replace the full recovery model during large bulk operations.

Table Requirements for Minimally Logging Bulk-Import Operations

Minimal logging requires that the target table meets the following conditions:

The table is not being replicated.

Table locking is specified (using TABLOCK). For table with clustered columnstore index, you don't need

TABLOCK for minimal logging. Additionally, only the data load into compressed rowgroups are minimally logged requiring a batchsize of 102400 or higher.

Although data insertions are not logged in the transaction log during a minimally logged bulk-import operation, the

Database Engine still logs extent allocations each time a new extent is allocated to the table.

Table is not a memory-optimized table.

Whether minimal logging can occur for a table also depends on whether the table is indexed and, if so, whether the table is empty:

If the table has no indexes, data pages are minimally logged.

If the table has no clustered index but has one or more nonclustered indexes, data pages are always minimally logged. How index pages are logged, however, depends on whether the table is empty:

If the table is empty, index pages are minimally logged.

If table is non-empty, index pages are fully logged.

If you start with an empty table and bulk import the data in multiple batches, both index and data pages are minimally logged for the first batch, but beginning with the second batch, only data pages are minimally logged.

If the table has a clustered index and is empty, both data and index pages are minimally logged. In contrast,

if a table has a btree based clustered index and is non-empty, data pages and index pages are both fully logged regardless of the recovery model. For tables with clustered columnstore index, the dataload into compressed rowgroup is always minimally logged independent of the table being empty or not when batchsize >= 102400.

If you start with an empty table rowstore table and bulk import the data in batches, both index and data pages are minimally logged for the first batch, but from the second batch onwards, only data pages are bulk logged.

When transactional replication is enabled, BULK INSERT operations are fully logged even under the Bulk Logged recovery model.

Related Tasks

View or Change the Recovery Model of a Database (SQL Server)

Prepare to Bulk Import Data (SQL Server)

3/24/2017 • 3 min to read • Edit Online

THIS TOPIC APPLIES TO:

SQL Server (starting with 2008) Azure SQL Database Azure SQL Data Warehouse

Parallel Data Warehouse

You can use the bcp command, BULK INSERT statement, or OPENROWSET(BULK) function to bulk import data from a data file only.

It is possible to write a custom application that bulk imports data from objects other than a text file. To bulk import data from memory buffers, use either the bcp extensions to the SQL Server Native Client (ODBC) application programming interface (API) or the OLE DB IRowsetFastLoad interface. To bulk import data from a C# data table, use the ADO.NET bulkcopy API, SqlBulkCopy.

Bulk importing data into a remote table is not supported.

Use the following guidelines when you bulk import data from a data file to an instance of Microsoft SQL Server:

Obtain required permissions for your user account.

The user account in which you use the bcp utility, the BULK INSERT statement, or the INSERT ... SELECT *

FROM OPENROWSET(BULK...) statement must have the required permissions on the table, which are assigned by the table owner. For more information about permissions that are required by each method, see bcp Utility , OPENROWSET (Transact-SQL) , and BULK INSERT (Transact-SQL) .

Use the bulk-logged recovery model.

This guideline is for a database that uses the full recovery model. The bulk-logged recovery model is useful when performing bulk operations into an unindexed table (a heap). Using bulk-logged recovery helps prevent the transaction log from running out of space because bulk-logged recovery does not perform log row inserts. For more information about the bulk-logged recovery model, see Recovery Models (SQL

Server) .

We recommend that you change the database to use the bulk-logged recovery model immediately before the bulk import operation. Immediately afterwards, you should reset the database to the full recovery model.

For more information, see View or Change the Recovery Model of a Database (SQL Server) .

more information about how to minimize logging during bulk import operations, see Prerequisites for Minimal

Logging in Bulk Import .

Back up after bulk importing data.

For a database that uses the simple recovery model, we recommend that you take a full or differential backup after the bulk-import operation finishes. For more information, see Create a Full Database Backup

(SQL Server) or Create a Differential Database Backup (SQL Server) .

For the bulk-logged recovery model or full recovery model, a log backup is enough. For more information, see Transaction Log Backups (SQL Server) .

Drop table indexes to improve performance for large bulk imports.

This guideline is for when you are importing a large amount of data compared to the amount of data that is already in the table. In this case, dropping the indexes from the table before you perform the bulk-import operation can significantly increase performance.

If you are loading a small amount of data compared to the amount of data already in the table, dropping the indexes is counterproductive. The time required to rebuild the indexes might be longer than the time saved during the bulkimport operation.

Find and remove hidden characters in the data file.

Many utilities and text editors display hidden characters, which are usually at the end of the data file. During a bulk-import operation, hidden characters in an ASCII data file can cause problems that cause an error of

"unexpected null found". Finding and removing all the hidden characters should help prevent this problem.

Prepare Data for Bulk Export or Import (SQL Server)

3/24/2017 • 5 min to read • Edit Online

THIS TOPIC APPLIES TO: SQL Server (starting with 2008) Azure SQL Database Azure SQL Data

Warehouse Parallel Data Warehouse

This section discusses the considerations involved in planning for bulk-export operations and the requirements for bulk-import operations.

If you are uncertain about how to format a data file for bulk importing, use the bcp utility to export data from the table into a data file. The formatting of each data field in this file shows the formatting required to bulk import data into the corresponding table column. Use the same data formatting for fields of your data file.

Data-File Format Considerations for Bulk Export

Before you perform a bulk-export operation by using the bcp command, consider the following:

When data is exported to a file, the bcp command creates the data file automatically by using the specified file name. If that file name is already in use, the data that is being bulk copied to the data file overwrites the existing contents of the file.

Bulk export from a table or view to a data file requires SELECT permission on the table or view that is being bulk copied.

Microsoft SQL Server can use parallel scans to retrieve data. Therefore, the table rows that are bulk exported in from an instance of SQL Server are not ordinarily guaranteed to be in any specific order in the data file. To make bulk-exported table rows appear in a specific order in the data file, use the queryout option to bulk export from a query, and specify an ORDER BY clause.

Data-File Format Requirements for Bulk Import

To import data from a data file, the file must meet the following basic requirements:

The data must be in row and column format.

The structure of the data file does not need to be identical to the structure of the SQL Server table because columns can be skipped or reordered during the bulk-import process.

The data in the data file must be in a supported format such as character or native format.

The data can be in character or native binary format including Unicode.

To import data by using a bcp command, BULK INSERT statement, or INSERT ... SELECT * FROM

OPENROWSET(BULK...) statement, the destination table must already exist.

Each field in the data file must be compatible with the corresponding column in the target table. For

example, an int field cannot be loaded into a datetime column. For more information, see Data Formats for

Bulk Import or Bulk Export (SQL Server) and Specify Data Formats for Compatibility when Using bcp (SQL

Server) .

To specify a subset of rows to import from a data file rather than the entire file, you can use a bcp command with the -F first_row switch and/or -L last_row switch. For more information, see bcp Utility .

To import data from data files with fixed-length or fixed-width fields, use a format file. For more

information, see XML Format Files (SQL Server)

Comma-separated value (CSV) files are not supported by SQL Server bulk-import operations. However, in some cases, a CSV file can be used as the data file for a bulk import of data into SQL Server. Note that the field terminator of a CSV file does not have to be a comma. To be usable as a data file for bulk import, a

CSV file must comply with the following restrictions:

Data fields never contain the field terminator.

Either none or all of the values in a data field are enclosed in quotation marks ("").

To bulk import data from a Microsoft FoxPro or Visual FoxPro table (.dbf) file or a Microsoft Excel worksheet (.xls) file, you would need to convert the data into a CSV file that complies to the preceding restrictions. The file extension will typically be .csv. You can then use the .csv file as a data file in a SQL Server bulk-import operation.

On 32-bit systems, it is possible to import CSV data into a SQL Server table without bulk-import optimizations by using OPENROWSET with the OLE DB Provider for Jet. Jet treats text files as tables, with the schema defined by a schema.ini file that is located in the same directory as the data source.

For a CSV data, one of the parameters in the schema.ini file would be "FORMAT=CSVDelimited". To use this solution, you would need to understand how the Jet Test IISAMm operations—its connection string syntax, schema.ini usage, registry setting options, and so on). The best sources of this information are Microsoft Access Help and Knowledge Base (KB) articles. For more information, see

Initializing the Text Data Source Driver , How To Use a SQL Server 7.0 Distributed Query with a Linked

Server to Secured Access Databases , HOW TO: Use Jet OLE DB Provider 4.0 to Connect to ISAM

Databases , and How To Open Delimited Text Files Using the Jet Provider's Text IIsam .

In addition, the bulk import of data from a data file into a table requires the following:

Users must have INSERT and SELECT permissions on the table. Users also need ALTER TABLE permission when they use options that require data definition language (DDL) operations, such as disabling constraints.

When you bulk import data by using BULK INSERT or INSERT ... SELECT * FROM OPENROWSET(BULK...), the data file must be accessible for read operations by either the security profile of the SQL Server process (if the user logs in using SQL Server provided login) or by the Microsoft Windows login that is used under delegated security. Additionally, the user must have ADMINISTER BULK OPERATIONS permission to read the file.

Bulk importing into a partitioned view is unsupported, and attempts to bulk import data into a partitioned view fail.

External Resources

How to import data from Excel to SQL Server

Change History

UPDATED CONTENT

Added information about using the OLE DB Provider for Jet to import CSV data.

HumanResources.myTeam Sample Table (SQL Server)

3/24/2017 • 1 min to read • Edit Online

Many of the code examples in Importing and Exporting Bulk Data

require a special-purpose test table named

myTeam. Before you can run the examples, you must create the myTeam table in the HumanResources schema of the AdventureWorks2012 database.

AdventureWorks2012 is one of the sample databases in SQL Server 2017.

The myTeam table is contains the following columns.

COLUMN DATA TYPE

EmployeeID smallint

NULLABILITY

Not null

DESCRIPTION

Primary key for the rows.

Employee ID of a member of my team.

Name of a member of my team.

Title the employee performs on my team.

Date and time the row was last updated. (Default)

Name

Title

Background nvarchar(50) nvarchar(50) nvarchar(50)

Not null

Nullable

Not null

To create HumanResources.myTeam

Use the following Transact-SQL statements:

--Create HumanResources.MyTeam:

USE AdventureWorks;

CREATE TABLE HumanResources.myTeam

(EmployeeID smallint NOT NULL,

Name nvarchar(50) NOT NULL,

Title nvarchar(50) NULL,

Background nvarchar(50) NOT NULL DEFAULT ''

);

To populate HumanResources.myTeam

Execute following

INSERT

statements to populate the table with two rows:

USE AdventureWorks;

INSERT INTO HumanResources.myTeam(EmployeeID,Name,Title,Background)

VALUES(77,'Mia Doppleganger','Administrative Assistant','Microsoft Office');

INSERT INTO HumanResources.myTeam(EmployeeID,Name,Title,Background)

VALUES(49,'Hirum Mollicat','I.T. Specialist','Report Writing and Data Mining');

These statements skip the fourth column,

Background

. This has a default value. Skipping this column causes this

INSERT

statement to leave this column blank.

Data Formats for Bulk Import or Bulk Export (SQL

Server)

3/24/2017 • 1 min to read • Edit Online

THIS TOPIC APPLIES TO: SQL Server (starting with 2008) Azure SQL Database Azure SQL Data

Warehouse Parallel Data Warehouse

SQL Server can accept data in character data format or native binary data format. Use character format when you move data between SQL Server and another application (such as Microsoft Excel) or another database server

(such as Oracle or SQL Server). You can use native format only when you transfer data between instances of SQL

Server.

In this Topic:

Data Formats for Bulk Import or Export

Use Character Format to Import or Export Data (SQL Server)

Data Formats for Bulk Import or Export

The following table indicates what data format is generally appropriate to use depending on how the data is represented and the source or target of the operation.

OPERATION NATIVE

Bulk transfers of data between multiple instances of SQL

Server by using a data file that does not contain any extended or doublebyte character set

(DBCS) characters.

Unless a format file is used, these tables must be identically defined.

Yes*

For sql_variant columns, use of native data format is best, because native data format preserves the metadata for each sql_variant value, unlike character or Unicode formats.

Yes

UNICODE NATIVE

—

CHARACTER

—

UNICODE CHARACTER

—

OPERATION NATIVE

Bulk transfers of data between multiple instances of SQL

Server by using a data file that contains extended or DBCS characters.

—

Bulk import of data from a text file that is generated by another program.

—

Bulk export of data to a text file that is to be used in another program.

—

Bulk transfers of data between multiple instances of SQL

Server by using a data file that contains

Unicode data and does not contain any extended or DBCS characters.

—

UNICODE NATIVE

Yes

—

CHARACTER

—

Yes

—

* Fastest method for the bulk export of data from SQL Server when using bcp.

Use Unicode Native Format to Import or Export Data (SQL Server)

Use Unicode Character Format to Import or Export Data (SQL Server)

Import Native and Character Format Data from Earlier Versions of SQL Server

Use Native Format to Import or Export Data (SQL

Server)

3/24/2017 • 7 min to read • Edit Online

Native format is recommended when you bulk transfer data between multiple instances of SQL Server using a data file that does not contain any extended/double-byte character set (DBCS) characters.

To bulk transfer data between multiple instances of SQL Server by using a data file that contains extended or DBCS

characters, you should use the Unicode native format. For more information, see Use Unicode Native Format to Import or

Export Data (SQL Server) .

Native format maintains the native data types of a database. Native format is intended for high-speed data transfer of data between SQL Server tables. If you use a format file, the source and target tables do not need to be identical. The data transfer involves two steps:

1. Bulk exporting the data from a source table into a data file

2. Bulk importing the data from the data file into the target table

The use of native format between identical tables avoids unnecessary conversion of data types to and from character format, saving time and space. To achieve the optimum transfer rate, however, few checks are performed regarding data formatting. To prevent problems with the loaded data, see the following restrictions list.

IN THIS TOPIC:

Restrictions

How bcp Handles Data in Native Format

Command Options for Native Format

● Using bcp and Native Format to Export Data

● Using bcp and Native Format to Import Data without a Format File

● Using bcp and Native Format to Import Data with a Non-XML Format File

● Using BULK INSERT and Native Format without a Format File

● Using BULK INSERT and Native Format with a Non-XML Format File

● Using OPENROWSET and Native Format with a Non-XML Format File

For information on using format files, see Format Files for Importing or Exporting Data (SQL Server)

Restrictions

To import data in native format successfully, ensure that:

The data file is in native format.

Either the target table must be compatible with the data file (having the correct number of columns, data type, length, NULL status, and so forth), or you must use a format file to map each field to its corresponding columns.

If you import data from a file that is mismatched with the target table, the import operation might succeed but the data values inserted into the target table are likely to be incorrect. This is because the data from the file is interpreted by using the format of the target table. Therefore, any mismatch results in the insertion of incorrect values. However, under no circumstances can such a mismatch cause logical or physical inconsistencies in the database.

A successful import will not corrupt the target table.

How bcp Handles Data in Native Format

This section discusses special considerations for how the bcp utility exports and imports data in native format.

Noncharacter data

The bcp utility uses the SQL Server internal binary data format to write noncharacter data from a table to a data file.

char or varchar data

At the beginning of each char or varchar field, bcp adds the prefix length.

When native mode is used, by default, the bcp utility converts characters from SQL Server to OEM characters before it copies them to a data file. The bcp utility converts characters from a data file to ANSI characters before it bulk imports them into a SQL Server table. During these conversions, extended character data can be lost. For extended characters, either use Unicode native format or specify a code page.

sql_variant data

If sql_variant data is stored as a SQLVARIANT in a native-format data file, the data maintains all of its characteristics. The metadata that records the data type of each data value is stored along with the data value. This metadata is used to re-create the data value with the same data type in a destination sql_variant column.

If the data type of the destination column is not sql_variant , each data value is converted to the data type of the destination column, following the normal rules of implicit data conversion. If an error occurs during data conversion, the current batch is rolled back. Any char and varchar values that are transferred between sql_variant columns may have code page conversion issues.

For more information about data conversion, see Data Type Conversion (Database Engine) .

Command Options for Native Format

You can import native format data into a table using bcp , BULK INSERT or INSERT ... SELECT * FROM

OPENROWSET(BULK...) . For a bcp command or BULK INSERT statement, you can specify the data format in the statement. For an INSERT ... SELECT * FROM OPENROWSET(BULK...) statement, you must specify the data format in a format file.

Native format is supported by the following command options:

COMMAND

bcp

OPTION

-n

DESCRIPTION

Causes the bcp utility to use the native data types of the data.*

BULK INSERT DATAFILETYPE ='native' Uses the native or wide native data types of the data. Note that

DATAFILETYPE is not needed if a format file specifies the data types.

OPENROWSET N/A Must use a format file

*To load native (-n) data to a format compatible with earlier versions of SQL Server clients, use the -V switch. For

more information, see Import Native and Character Format Data from Earlier Versions of SQL Server

Alternatively, you can specify formatting on a per-field basis in a format file. For more information, see Format Files for

original format that is supported by earlier versions of SQL Server. Please review Non-XML Format Files (SQL

Example Test Conditions

The examples in this topic are based on the table, and format file defined below.

The script below creates a test database, a table named myNative

and populates the table with some initial values. Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

CREATE DATABASE TestDatabase;

USE TestDatabase;

CREATE TABLE dbo.myNative (

PersonID smallint NOT NULL,

FirstName varchar(25) NOT NULL,

LastName varchar(30) NOT NULL,

BirthDate date,

AnnualSalary money

);

-- Populate table

INSERT TestDatabase.dbo.myNative

VALUES

(1, 'Anthony', 'Grosse', '1980-02-23', 65000.00),

(2, 'Alica', 'Fatnowna', '1963-11-14', 45000.00),

(3, 'Stella', 'Rossenhain', '1992-03-02', 120000.00);

-- Review Data

SELECT * FROM TestDatabase.dbo.myNative;

SQL Server support two types of format file: non-XML format and XML format. The non-XML format is the

Server) for detailed information. The following command will use the bcp utility to generate a non-xml format

file, myNative.fmt

, based on the schema of myNative

. To use a bcp command to create a format file, specify the

format argument and use nul instead of a data-file path. The format option also requires the -f option. In addition, for this example, the qualifier c is used to specify character data, and T is used to specify a trusted connection using integrated security. At a command prompt, enter the following commands: bcp TestDatabase.dbo.myNative format nul -f D:\BCP\myNative.fmt -T -n

REM Review file

Notepad D:\BCP\myNative.fmt

Ensure your non-XML format file ends with a carriage return\line feed. Otherwise you will likely receive the following error message:

SQLState = S1000, NativeError = 0

Error = [Microsoft][ODBC Driver 13 for SQL Server]I/O error while reading BCP format file

Examples

The examples below use the database, and format files created above.

-n switch and OUT command. Note: the data file created in this example will be used in all subsequent examples.

At a command prompt, enter the following commands: bcp TestDatabase.dbo.myNative OUT D:\BCP\myNative.bcp -T -n

REM Review results

NOTEPAD D:\BCP\myNative.bcp

-n switch and IN command. At a command prompt, enter the following commands:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myNative;"

REM Import data bcp TestDatabase.dbo.myNative IN D:\BCP\myNative.bcp -T -n

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myNative;"

-n and -f switches and IN command. At a command prompt, enter the following commands:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myNative;"

REM Import data bcp TestDatabase.dbo.myNative IN D:\BCP\myNative.bcp -f D:\BCP\myNative.fmt -T

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myNative;"

DATAFILETYPE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myNative; -- for testing

BULK INSERT TestDatabase.dbo.myNative

FROM 'D:\BCP\myNative.bcp'

WITH (

DATAFILETYPE = 'native'

);

-- review results

SELECT * FROM TestDatabase.dbo.myNative;

FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myNative; -- for testing

BULK INSERT TestDatabase.dbo.myNative

FROM 'D:\BCP\myNative.bcp'

WITH (

FORMATFILE = 'D:\BCP\myNative.fmt'

);

-- review results

SELECT * FROM TestDatabase.dbo.myNative;

FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myNative; -- for testing

INSERT INTO TestDatabase.dbo.myNative

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myNative.bcp',

FORMATFILE = 'D:\BCP\myNative.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myNative;

Related Tasks

To use data formats for bulk import or bulk export

Import Native and Character Format Data from Earlier Versions of SQL Server

Use Character Format to Import or Export Data (SQL Server)

Use Unicode Character Format to Import or Export Data (SQL Server)

Use Unicode Native Format to Import or Export Data (SQL Server)

Use Character Format to Import or Export Data

(SQL Server)

3/24/2017 • 6 min to read • Edit Online

Character format is recommended when you bulk export data to a text file that is to be used in another program or when you bulk import data from a text file that is generated by another program.

Character format uses the character data format for all columns. Storing information in character format is useful when the data is used with another program, such as a spreadsheet, or when the data needs to be copied into an instance of SQL Server from another database vendor such as Oracle.

When you bulk transfer data between instances of Microsoft SQL Server and the data file contains Unicode character data

but not any extended or DBCS characters, use the Unicode character format. For more information, see Use Unicode

Character Format to Import or Export Data (SQL Server) .

IN THIS TOPIC:

Considerations for Using Character Format

Command Options for Character Format

● Using bcp and Character Format to Export Data

● Using bcp and Character Format to Import Data without a Format File

● Using bcp and Character Format to Import Data with a Non-XML Format File

● Using BULK INSERT and Character Format without a Format File

● Using BULK INSERT and Character Format with a Non-XML Format File

● Using OPENROWSET and Character Format with a Non-XML Format File

Considerations for Using Character Format

When using character format, consider the following:

By default, the bcp utility separates the character-data fields with the tab character and terminates the records with the newline character. For information about how to specify alternative terminators, see

For more information, see Import Native and Character Format Data from Earlier Versions of SQL Server

By default, before the bulk export or import of character-mode data, the following conversions are performed:

DIRECTION OF BULK OPERATION

Export

Import

CONVERSION

Converts data to character representation. If explicitly requested, the data is converted to the requested code page for character columns. If no code page is specified, the character data is converted by using the OEM code page of the client computer.

Converts character data to native representation, when necessary, and translates the character data from the client's code page to the code page of the target column(s).

To prevent loss of extended characters during conversion, either use Unicode character format or specify a code page.

Any sql_variant data that is stored in a character-format file is stored without metadata. Each data value is converted to char format, according to the rules of implicit data conversion. When imported into a sql_variant column, the data is imported as char . When imported into a column with a data type other than sql_variant , the data is converted from char by using implicit conversion. For more information about data conversion, see Data Type Conversion (Database Engine) .

The bcp utility exports money values as character-format data files with four digits after the decimal point and without any digit-grouping symbols such as comma separators. For example, a money column that contains the value 1,234,567.123456 is bulk exported to a data file as the character string 1234567.1235.

Command Options for Character Format

You can import character format data into a table using bcp , BULK INSERT or INSERT ... SELECT * FROM

Character format is supported by the following command options:

COMMAND OPTION DESCRIPTION

bcp

BULK INSERT

-c

DATAFILETYPE ='char'

Causes the bcp utility to use character data.*

Use character format when bulk importing data.

Must use a format file OPENROWSET N/A

*To load character (-c) data to a format compatible with earlier versions of SQL Server clients, use the -V switch.

Alternatively, you can specify formatting on a per-field basis in a format file. For more information, see Format Files for

original format that is supported by earlier versions of SQL Server. Please review Non-XML Format Files (SQL

Example Test Conditions

The examples in this topic are based on the table, and format file defined below.

The script below creates a test database, a table named myChar

and populates the table with some initial values.

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

CREATE DATABASE TestDatabase;

USE TestDatabase;

CREATE TABLE dbo.myChar (

PersonID smallint NOT NULL,

FirstName varchar(25) NOT NULL,

LastName varchar(30) NOT NULL,

BirthDate date,

AnnualSalary money

);

-- Populate table

INSERT TestDatabase.dbo.myChar

VALUES

(1, 'Anthony', 'Grosse', '1980-02-23', 65000.00),

(2, 'Alica', 'Fatnowna', '1963-11-14', 45000.00),

(3, 'Stella', 'Rossenhain', '1992-03-02', 120000.00);

-- Review Data

SELECT * FROM TestDatabase.dbo.myChar;

SQL Server support two types of format file: non-XML format and XML format. The non-XML format is the

Server) for detailed information. The following command will use the bcp utility to generate a non-xml format

file, myChar.fmt

, based on the schema of myChar

. To use a bcp command to create a format file, specify the

format argument and use nul instead of a data-file path. The format option also requires the -f option. In addition, for this example, the qualifier c is used to specify character data, and T is used to specify a trusted connection using integrated security. At a command prompt, enter the following command: bcp TestDatabase.dbo.myChar format nul -f D:\BCP\myChar.fmt -T -c

REM Review file

Notepad D:\BCP\myChar.fmt

Ensure your non-XML format file ends with a carriage return\line feed. Otherwise you will likely receive the following error message:

SQLState = S1000, NativeError = 0

Error = [Microsoft][ODBC Driver 13 for SQL Server]I/O error while reading BCP format file

Examples

The examples below use the database, and format files created above.

-c switch and OUT command. Note: the data file created in this example will be used in all subsequent examples.

At a command prompt, enter the following command:

bcp TestDatabase.dbo.myChar OUT D:\BCP\myChar.bcp -T -c

REM Review results

NOTEPAD D:\BCP\myChar.bcp

-c switch and IN command. At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myChar;"

REM Import data bcp TestDatabase.dbo.myChar IN D:\BCP\myChar.bcp -T -c

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myChar;"

-c and -f switches and IN command. At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myChar;"

REM Import data bcp TestDatabase.dbo.myChar IN D:\BCP\myChar.bcp -f D:\BCP\myChar.fmt -T

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myChar;"

DATAFILETYPE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myChar; -- for testing

BULK INSERT TestDatabase.dbo.myChar

FROM 'D:\BCP\myChar.bcp'

WITH (

DATAFILETYPE = 'Char'

);

-- review results

SELECT * FROM TestDatabase.dbo.myChar;

FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myChar; -- for testing

BULK INSERT TestDatabase.dbo.myChar

FROM 'D:\BCP\myChar.bcp'

WITH (

FORMATFILE = 'D:\BCP\myChar.fmt'

);

-- review results

SELECT * FROM TestDatabase.dbo.myChar;

FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myChar; -- for testing

INSERT INTO TestDatabase.dbo.myChar

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myChar.bcp',

FORMATFILE = 'D:\BCP\myChar.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myChar;

Related Tasks

To use data formats for bulk import or bulk export

Import Native and Character Format Data from Earlier Versions of SQL Server

Use Native Format to Import or Export Data (SQL Server)

Use Unicode Character Format to Import or Export Data (SQL Server)

Use Unicode Native Format to Import or Export Data (SQL Server)

Use Unicode Native Format to Import or Export

Data (SQL Server)

3/24/2017 • 5 min to read • Edit Online

Unicode native format is helpful when information must be copied from one Microsoft SQL Server installation to another. The use of native format for noncharacter data saves time, eliminating unnecessary conversion of data types to and from character format. The use of Unicode character format for all character data prevents loss of any extended characters during bulk transfer of data between servers using different code pages. A data file in

Unicode native format can be read by any bulk-import method.

Unicode native format is recommended for the bulk transfer of data between multiple instances of SQL Server by using a data file that contains extended or DBCS characters. For noncharacter data, Unicode native format uses native (database) data types. For character data, such as char , nchar , varchar , nvarchar , text , varchar(max) , nvarchar(max) , and ntext , the Unicode native format uses Unicode character data format.

The sql_variant data that is stored as a SQLVARIANT in a Unicode native-format data file operates in the same manner as it does in a native-format data file, except that char and varchar values are converted to nchar and nvarchar , which doubles the amount of storage required for the affected columns. The original metadata is preserved, and the values are converted back to their original char and varchar data type when bulk imported into a table column.

IN THIS TOPIC:

Command Options for Unicode Native Format

● Using bcp and Unicode Native Format to Export Data

● Using bcp and Unicode Native Format to Import Data without a Format File

● Using bcp and Unicode Native Format to Import Data with a Non-XML Format File

● Using BULK INSERT and Unicode Native Format without a Format File

● Using BULK INSERT and Unicode Native Format with a Non-XML Format File

● Using OPENROWSET and Unicode Native Format with a Non-XML Format File

Alternatively, you can specify formatting on a per-field basis in a format file. For more information, see Format Files for

Command Options for Unicode Native Format

You can import Unicode native format data into a table using bcp , BULK INSERT or INSERT ... SELECT * FROM

Unicode native format is supported by the following command options:

COMMAND

bcp

OPTION

-N

DATAFILETYPE ='widenative'

N/A

DESCRIPTION

Causes the bcp utility to use the

Unicode native format, which uses native (database) data types for all noncharacter data and Unicode character data format for all character

(char, nchar, varchar, nvarchar, text, and ntext) data.

Uses Unicode native format when bulk importing data.

Must use a format file

BULK INSERT

OPENROWSET

original format that is supported by earlier versions of SQL Server. Please review Non-XML Format Files (SQL

Example Test Conditions

The examples in this topic are based on the table, and format file defined below.

The script below creates a test database, a table named myWidenative

and populates the table with some initial values. Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

CREATE DATABASE TestDatabase;

USE TestDatabase;

CREATE TABLE dbo.myWidenative (

PersonID smallint NOT NULL,

FirstName nvarchar(25) NOT NULL,

LastName nvarchar(30) NOT NULL,

BirthDate date,

AnnualSalary money

);

-- Populate table

INSERT TestDatabase.dbo.myWidenative

VALUES

(1, N'ϴAnthony', N'Grosse', '02-23-1980', 65000.00),

(2, N'

❤

Alica', N'Fatnowna', '11-14-1963', 45000.00),

(3, N'

☎

Stella', N'Rossenhain', '03-02-1992', 120000.00);

-- Review Data

SELECT * FROM TestDatabase.dbo.myWidenative;

SQL Server support two types of format file: non-XML format and XML format. The non-XML format is the

Server) for detailed information. The following command will use the bcp utility to generate a non-xml format

file, myWidenative.fmt

, based on the schema of myWidenative

trusted connection using integrated security. At a command prompt, enter the following commands: bcp TestDatabase.dbo.myWidenative format nul -f D:\BCP\myWidenative.fmt -T -N

REM Review file

Notepad D:\BCP\myWidenative.fmt

Ensure your non-XML format file ends with a carriage return\line feed. Otherwise you will likely receive the following error message:

SQLState = S1000, NativeError = 0

Error = [Microsoft][ODBC Driver 13 for SQL Server]I/O error while reading BCP format file

Examples

The examples below use the database, and format files created above.

-N switch and OUT command. Note: the data file created in this example will be used in all subsequent examples. At a command prompt, enter the following commands: bcp TestDatabase.dbo.myWidenative OUT D:\BCP\myWidenative.bcp -T -N

REM Review results

NOTEPAD D:\BCP\myWidenative.bcp

-N switch and IN command. At a command prompt, enter the following commands:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myWidenative;"

REM Import data bcp TestDatabase.dbo.myWidenative IN D:\BCP\myWidenative.bcp -T -N

REM Review results is SSMS

-N and -f switches and IN command. At a command prompt, enter the following commands:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myWidenative;"

REM Import data bcp TestDatabase.dbo.myWidenative IN D:\BCP\myWidenative.bcp -f D:\BCP\myWidenative.fmt -T

REM Review results is SSMS

DATAFILETYPE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myWidenative; -- for testing

BULK INSERT TestDatabase.dbo.myWidenative

FROM 'D:\BCP\myWidenative.bcp'

WITH (

DATAFILETYPE = 'widenative'

);

-- review results

SELECT * FROM TestDatabase.dbo.myWidenative;

FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myWidenative; -- for testing

BULK INSERT TestDatabase.dbo.myWidenative

FROM 'D:\BCP\myWidenative.bcp'

WITH (

FORMATFILE = 'D:\BCP\myWidenative.fmt'

);

-- review results

SELECT * FROM TestDatabase.dbo.myWidenative;

FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myWidenative; -- for testing

INSERT INTO TestDatabase.dbo.myWidenative

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myWidenative.bcp',

FORMATFILE = 'D:\BCP\myWidenative.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myWidenative;

Related Tasks

To use data formats for bulk import or bulk export

Import Native and Character Format Data from Earlier Versions of SQL Server

Use Character Format to Import or Export Data (SQL Server)

Use Native Format to Import or Export Data (SQL Server)

Use Unicode Character Format to Import or Export Data (SQL Server)

Use Unicode Character Format to Import or Export

Data (SQL Server)

3/24/2017 • 8 min to read • Edit Online

Unicode character format is recommended for bulk transfer of data between multiple instances of SQL Server by using a data file that contains extended/DBCS characters. The Unicode character data format allows data to be exported from a server by using a code page that differs from the code page used by the client that is performing the operation. In such cases, use of Unicode character format has the following advantages:

If the source and destination data are Unicode data types, use of Unicode character format preserves all of the character data.

If the source and destination data are not Unicode data types, use of Unicode character format minimizes the loss of extended characters in the source data that cannot be represented at the destination.

IN THIS TOPIC:

Considerations for Using Unicode Character Format

Special Considerations for Using Unicode Character Format, bcp, and a Format File

Command Options for Unicode Character Format

● Using bcp and Unicode Character Format to Export Data

● Using bcp and Unicode Character Format to Import Data without a Format File

● Using bcp and Unicode Character Format to Import Data with a Non-XML Format File

● Using BULK INSERT and Unicode Character Format without a Format File

● Using BULK INSERT and Unicode Character Format with a Non-XML Format File

● Using OPENROWSET and Unicode Character Format with a Non-XML Format File

Considerations for Using Unicode Character Format

When using Unicode character format, consider the following:

Don’t use a format file. An example of this workaround is provided below, see Using bcp and Unicode

The sql_variant data that is stored in a Unicode character-format data file operates in the same way it operates in a character-format data file, except that the data is stored as nchar instead of char data. For more information about character format, see Collation and Unicode Support .

Special Considerations for Using Unicode Character Format, bcp, and

a Format File

Unicode character format data files follow the conventions for Unicode files. The first two bytes of the file are hexadecimal numbers, 0xFFFE. These bytes serve as byte-order marks (BOM), specifying whether the high-order byte is stored first or last in the file. The bcp Utility may misinterpret the BOM and cause part of your import process to fail; you may receive an error message similar as follows:

Starting copy...

SQLState = 22005, NativeError = 0

Error = [Microsoft][ODBC Driver 13 for SQL Server]Invalid character value for cast specification

The BOM may be misinterpreted under the following conditions:

The bcp Utility is used and the -w switch is used to indicate Unicode character

A format file is used

The first field in the data file is non-character

Consider whether any of the following workarounds may be available for your specific situation:

Character Format to Import Data without a Format File ,

Use the -c switch instead of -w,

Re-export the data using a native format,

Use BULK INSERT or OPENROWSET . Examples of these workaround are provided below, see Using BULK

INSERT and Unicode Character Format with a Non-XML Format File and Using OPENROWSET and

Unicode Character Format with a Non-XML Format File ,

Manually insert first record in destination table and then use -F 2 switch to have import start on second record,

Manually insert dummy first record in data file and then use -F 2 switch to have import start on second

record. An example of this workaround is provided below, see Using bcp and Unicode Character Format to Import Data with a Non-XML Format File ,

Use a staging table where the first column is a character data type, or

Re-export the data and change the data field order so that the first data field will be character. Then use a

format file to remap the data field to the actual order in the table. For an example, see Use a Format File to

Map Table Columns to Data-File Fields (SQL Server) .

Command Options for Unicode Character Format

You can import Unicode character format data into a table using bcp , BULK INSERT or INSERT ... SELECT * FROM

Unicode character format is supported by the following command options:

COMMAND OPTION DESCRIPTION

bcp

-w

Uses the Unicode character format.

COMMAND

BULK INSERT

OPENROWSET

OPTION

DATAFILETYPE ='widechar'

N/A

DESCRIPTION

Uses Unicode character format when bulk importing data.

Must use a format file

Alternatively, you can specify formatting on a per-field basis in a format file. For more information, see Format Files for

original format that is supported by earlier versions of SQL Server. Please review Non-XML Format Files (SQL

Example Test Conditions

The examples in this topic are based on the table, and format file defined below.

The script below creates a test database, a table named myWidechar

and populates the table with some initial values. Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

CREATE DATABASE TestDatabase;

USE TestDatabase;

CREATE TABLE dbo.myWidechar (

PersonID smallint NOT NULL,

FirstName nvarchar(25) NOT NULL,

LastName nvarchar(30) NOT NULL,

BirthDate date,

AnnualSalary money

);

-- Populate table

INSERT TestDatabase.dbo.myWidechar

VALUES

(1, N'ϴAnthony', N'Grosse', '02-23-1980', 65000.00),

(2, N'

❤

Alica', N'Fatnowna', '11-14-1963', 45000.00),

(3, N'

☎

Stella', N'Rossenhain', '03-02-1992', 120000.00);

-- Review Data

SELECT * FROM TestDatabase.dbo.myWidechar;

SQL Server support two types of format file: non-XML format and XML format. The non-XML format is the

Server) for detailed information. The following command will use the bcp utility to generate a non-xml format

file, myWidechar.fmt

, based on the schema of myWidechar

. To use a bcp command to create a format file, specify the format argument and use nul instead of a data-file path. The format option also requires the -f option. In addition, for this example, the qualifier c is used to specify character data, and T is used to specify a trusted connection using integrated security. At a command prompt, enter the following commands: bcp TestDatabase.dbo.myWidechar format nul -f D:\BCP\myWidechar.fmt -T -w

REM Review file

Notepad D:\BCP\myWidechar.fmt

Ensure your non-XML format file ends with a carriage return\line feed. Otherwise you will likely receive the following error message:

SQLState = S1000, NativeError = 0

Error = [Microsoft][ODBC Driver 13 for SQL Server]I/O error while reading BCP format file

Examples

The examples below use the database, and format files created above.

-w switch and OUT command. Note: the data file created in this example will be used in all subsequent examples.

At a command prompt, enter the following commands: bcp TestDatabase.dbo.myWidechar OUT D:\BCP\myWidechar.bcp -T -w

REM Review results

NOTEPAD D:\BCP\myWidechar.bcp

-w switch and IN command. At a command prompt, enter the following commands:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myWidechar;"

REM Import data bcp TestDatabase.dbo.myWidechar IN D:\BCP\myWidechar.bcp -T -w

REM Review results is SSMS

-w and -f switches and IN command. A workaround will need to be used since this example involves bcp, a

format file, Unicode character, and the first data field in the data file is non-character. See Special Considerations for Using Unicode Character Format, bcp, and a Format File , above. The data file myWidechar.bcp

will be altered

by adding an additional record as a “dummy” record which will then be skipped with the

-F 2

switch.

At a command prompt, enter the following commands and follow the modification steps:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myWidechar;"

REM Open data file

Notepad D:\BCP\myWidechar.bcp

REM Copy first record and then paste as new first record. This additional record is the "dummy" record.

REM Close file.

REM Import data instructing bcp to skip dummy record with the -F 2 switch.

bcp TestDatabase.dbo.myWidechar IN D:\BCP\myWidechar.bcp -f D:\BCP\myWidechar.fmt -T -F 2

REM Review results is SSMS

REM Return data file to original state for usage in other examples bcp TestDatabase.dbo.myWidechar OUT D:\BCP\myWidechar.bcp -T -w

DATAFILETYPE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myWidechar; -- for testing

BULK INSERT TestDatabase.dbo.myWidechar

FROM 'D:\BCP\myWidechar.bcp'

WITH (

DATAFILETYPE = 'widechar'

);

-- review results

SELECT * FROM TestDatabase.dbo.myWidechar;

FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myWidechar; -- for testing

BULK INSERT TestDatabase.dbo.myWidechar

FROM 'D:\BCP\myWidechar.bcp'

WITH (

FORMATFILE = 'D:\BCP\myWidechar.fmt'

);

-- review results

SELECT * FROM TestDatabase.dbo.myWidechar;

FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

TRUNCATE TABLE TestDatabase.dbo.myWidechar; -- for testing

INSERT INTO TestDatabase.dbo.myWidechar

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myWidechar.bcp',

FORMATFILE = 'D:\BCP\myWidechar.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myWidechar;

Related Tasks

To use data formats for bulk import or bulk export

Import Native and Character Format Data from Earlier Versions of SQL Server

Use Character Format to Import or Export Data (SQL Server)

Use Native Format to Import or Export Data (SQL Server)

Use Unicode Native Format to Import or Export Data (SQL Server)

Import Native and Character Format Data from

Earlier Versions of SQL Server

3/24/2017 • 2 min to read • Edit Online

In SQL Server 2017, you can use bcp to import native and character format data from SQL Server 2000, SQL

Server 2005, SQL Server 2008, SQL Server 2008 R2, or SQL Server 2012 by using the -V switch. The -V switch causes SQL Server 2017 to use data types from the specified earlier version of SQL Server, and the data file format are the same as the format in that earlier version.

To specify an earlier SQL Server version for a data file, use the -V switch with one of the following qualifiers:

SQL SERVER VERSION

SQL Server 2000

SQL Server 2005

SQL Server 2008

SQL Server 2012

QUALIFIER

-V80

-V90

-V100

-V 110

Interpretation of Data Types

SQL Server 2005 and later versions have support for some new types. When you want to import a new data type into an earlier SQL Server version, the data type must be stored in a format that readable by the older bcp clients. The following table summarizes how the new data types are converted for compatibility with the earlier versions of SQL Server.

NEW DATA TYPES IN SQL

SERVER 2005 bigint

COMPATIBLE DATA TYPES IN

VERSION 6X decimal

COMPATIBLE DATA TYPES IN

VERSION 70 decimal

COMPATIBLE DATA TYPES IN

VERSION 80 sql_variant varchar(max) nvarchar(max) text text ntext nvarchar(4000) text ntext

text ntext varbinary(max)

XML

UDT**

image ntext image

*This type is natively supported.

**UDT indicates a user defined type.

image ntext image image ntext image

Exporting using –V 80

When you bulk export data by using the –V80 switch, nvarchar(max), varchar(max), varbinary(max), XML, and UDT data in native mode are stored with a 4-byte prefix, like text, image, and ntext data, rather than with an 8-byte prefix, which is the default for SQL Server 2005 and later versions.

Copying Date Values

bcp uses the ODBC bulk copy API. Therefore, to import date values into SQL Server, bcp uses the ODBC date format (yyyy-mm-dd hh:mm:ss[.f...]).

The bcp command exports character format data files using the ODBC default format for datetime and

smalldatetime values. For example, a datetime column containing the date

12 Aug 1998

is bulk copied to a data file as the character string

1998-08-12 00:00:00.000

When importing data into a smalldatetime field using bcp, be sure the value for seconds is 00.000; otherwise the operation will fail. The smalldatetime data type only holds values to the nearest minute. BULK INSERT and INSERT ...

SELECT * FROM OPENROWSET(BULK...) will not fail in this instance but will truncate the seconds value.

Related Tasks

To use data formats for bulk import or bulk export

Use Character Format to Import or Export Data (SQL Server)

Use Native Format to Import or Export Data (SQL Server)

Use Unicode Character Format to Import or Export Data (SQL Server)

Use Unicode Native Format to Import or Export Data (SQL Server)

Specify Data Formats for Compatibility when Using bcp (SQL Server)

3/24/2017 • 5 min to read • Edit Online

This topic describes the data-format attributes, field-specific prompts, and storing field-by-field data in a non-xml format file of the SQL Serverbcp command. Understanding these can be helpful when you bulk export SQL

Server data for bulk import into another program, such as another database program. The default data formats

(native, character, or Unicode) in the source table might be incompatible with the data layout expected by the other program If an incompatibility exists, when you export the data, you must describe the data layout.

If you are unfamiliar with data formats for importing or exporting data, see Data Formats for Bulk Import or Bulk Export

are exported as their base types. For more information, see Specify File Storage Type by Using bcp (SQL

bcp Data-Format Attributes

The bcp command allows you to specify the structure of each field in a data file in terms of the following dataformat attributes:

File storage type

The file storage type describes how data is stored in the data file. Data can be exported to a data file as its database table type (native format), in its character representation (character format), or as any data type where implicit conversion is supported; for example, copying a smallint as an int. User-defined data types

characters are called length prefix characters. For more information, see Specify Prefix Length in Data Files by Using bcp (SQL Server) .

Prefix length

To provide the most compact file storage for the bulk export of data in native format to a data file, the bcp command precedes each field with one or more characters that indicates the length of the field. These

Field length

information, see Specify Field Length by Using bcp (SQL Server)

Field terminator

For character data fields, optional terminating characters allow you to mark the end of each field in a data file (using a field terminator) and the end of each row (using a row terminator). Terminating characters are one way to indicate to programs reading the data file where one field or row ends and another begins. For

more information, see Specify Field and Row Terminators (SQL Server)

Overview of the Field-Specific Prompts

If an interactive bcp command contains the in or out option but does not also contain either the format file

switch (-f) or a data-format switch (-n, -c, -w, or -N), each column in the source or target table, the command prompts for each of the preceding attributes, in turn. In each prompt, the bcp command provides a default value based on the SQL Server data type of the table column. Accepting the default value for all of the prompts produces the same result as specifying native format (-n) on the command line. Each prompt displays a default value in brackets: [default]. Pressing ENTER accepts the displayed default. To specify a value other than the default, enter the new value at the prompt.

The following example uses the bcp command to bulk export data from the

HumanResources.myTeam

table interactively to the myTeam.txt

file. Before you can run the example, you must create this table. For information

about the table and how to create it, see HumanResources.myTeam Sample Table (SQL Server)

The command specifies neither a format file nor a data type, causing bcp to prompt for data-format information.

At the Microsoft Windows command prompt, enter: bcp AdventureWorks.HumanResources.myTeam out myTeam.txt -T

For each column, bcp prompts for field-specific values. The following example shows the field-specific prompts for the

EmployeeID

and

Name

columns of the table, and suggests the default file storage type (the native format) for each column. The prefix lengths of the

EmployeeID

and

Name

column are 0 and 2, respectively. The user specifies a comma (

) as the terminator of each field.

Enter the file storage type of field EmployeeID [smallint]:

Enter prefix-length of field EmployeeID [0]:

Enter field terminator [none]:,

Enter the file storage type of field Name [nvarchar]:

Enter prefix length of field Name [2]:

Enter field terminator [none]:,

Equivalent prompts (as needed) are displayed for each of the table columns in order.

Storing Field-by-Field Data in a Non-XML Format File

After all of the table columns are specified, the bcp command prompts you to optionally generate a non-XML format file that stores the field-by-field information just supplied (see the preceding example). If you choose to generate a format file, you can whenever you export data out of that table or import like-structured data into SQL

Server.

You can use the format file to bulk import data from the data file into an instance of SQL Server or to bulk export data from

the table, without needing to respecify the format. For more information, see Format Files for Importing or Exporting Data

The following example creates a non-XML format file named myFormatFile.fmt

Do you want to save this format information in a file? [Y/n] y

Host filename: [bcp.fmt]myFormatFile.fmt

The default name for the format file is bcp.fmt, but you can specify a different file name if you choose.

For a data file that uses a single data format for its file-storage type, such as character or native format, you can quickly create a format file without exporting or importing data by using the format option. This approach has the advantages of being easy and of allowing you to create either an XML format file or a non-XML format file. For more information, see

Specify Prefix Length in Data Files by Using bcp (SQL Server)

Specify Field Length by Using bcp (SQL Server)

for each field in a non-XML format file. For more information on non-XML format files, see Non-XML Format Files

Specify File Storage Type by Using bcp (SQL Server)

3/24/2017 • 3 min to read • Edit Online

The bcp Prompt for File Storage Type

If an interactive bcp command contains the in or out option without either the format file switch (-f) or a dataformat switch (-n, -c, -w, or -N), the command prompts for the file storage type of each data field, as follows:

Enter the file storage type of field <field_name> [<default>]:

Your response to this prompt depends on the task you perform, as follows:

To bulk export data from an instance of Microsoft SQL Server to a data file in the most compact storage possible (native data format), accept the default file storage types that are provided by bcp. For a list of the native file storage types, see "Native File Storage Types," later in this topic.

To bulk export data from an instance of SQL Server to a data file in character format, specify char as the file storage type for all columns in the table.

To bulk import data to an instance of SQL Server from a data file, specify the file storage type as char for types stored in character format and, for data stored in native data type format, specify one of the file storage types, as appropriate:

FILE STORAGE TYPE

char*

ENTER AT COMMAND PROMPT

c[har]

varchar c[har] nchar nvarchar

text\

ntext2 binary varbinary

image\

datetime smalldatetime w w

T[ext]

W x x

I[mage]

d[ate]

FILE STORAGE TYPE time date datetime2 datetimeoffset decimal numeric float real

Int bigint smallint tinyint money smallmoney n f[loat] r i[nt]

B[igint] s[mallint] t[inyint] m[oney]

M do n de d2

ENTER AT COMMAND PROMPT te bit uniqueidentifier sql_variant timestamp

UDT (a user-defined data type)

XML

X b[it] u

V[ariant] x

*The interaction of field length, prefix length, and terminators determines the amount of storage space that is allocated in a data file for noncharacter data that is exported as the char file storage type.

** The ntext, text, and image data types will be removed in a future version of SQL Server. In new development work, avoid using these data types, and plan to modify applications that currently use them.

Use nvarchar(max), varchar(max), and varbinary(max) instead.

Native File Storage Types

Each native file storage type is recorded in the format file as a corresponding host file data type.

real int bigint smallint tinyint money smallmoney bit uniqueidentifier sql_variant timestamp

FILE STORAGE TYPE

char*

varchar nchar nvarchar

text\

ntext\

binary varbinary

image\

datetime smalldatetime decimal numeric float

SQLFLT4

SQLINT

SQLBIGINT

SQLSMALLINT

SQLTINYINT

SQLMONEY

SQLMONEY4

SQLBIT

SQLUNIQUEID

SQLVARIANT

SQLBINARY

HOST FILE DATA TYPE

SQLCHAR

SQLNCHAR

SQLCHAR

SQLNCHAR

SQLBINARY

SQLDATETIME

SQLDATETIM4

SQLDECIMAL

SQLNUMERIC

SQLFLT8

FILE STORAGE TYPE

UDT (a user-defined data type)

HOST FILE DATA TYPE

SQLUDT

*Data files that are stored in character format use char as the file storage type. Therefore, for character data files,

SQLCHAR is the only data type that appears in a format file.

**You cannot bulk import data into text, ntext, and image columns that have DEFAULT values.

Additional Considerations for File Storage Types

When you bulk export data from an instance of SQL Server to a data file:

You can always specify char as the file storage type.

If you enter a file storage type that represents an invalid implicit conversion, bcp fails; for example, though you can specify int for smallint data, if you specify smallint for int data, overflow errors result.

When noncharacter data types such as float, money, datetime, or int are stored as their database types, the data is written to the data file in the SQL Server native format.

After you interactively specify all of the fields in a bcp command, the command prompts you save your responses

field in a non-XML format file. For more information about non-XML format files, see Non-XML Format Files (SQL Server)

Specify Prefix Length in Data Files by Using bcp

(SQL Server)

3/24/2017 • 3 min to read • Edit Online

The bcp Prompt for Prefix Length

Enter prefix length of field <field_name> [<default>]:

If you specify 0, bcp prompts you for either the length of the field (for a character data type) or a field terminator

(for a native non-character type).

After you interactively specify all of the fields in a bcp command, the command prompts you save your responses for each

Overview of Prefix Length

To store the prefix length of a field, you need enough bytes to represent the maximum length of the field. The number of bytes that are required also depends upon the file storage type, the nullability of a column, and whether the data is being stored in the data file in its native or character format. For example, a text or image data type requires four prefix characters to store the field length, but a varchar data type requires two characters.

In the data file, these length-prefix characters are stored in the internal binary data format of SQL Server.

When you use native format, use length prefixes rather than field terminators. Native format data might conflict with terminators because a native-format data file is stored in the SQL Server internal binary data format.

Prefix Lengths for Bulk Export

The default value that is provided at the prefix-length prompt when you export a field indicates the most efficient prefix length for the field.

Null values are represented as an empty field. To indicate that the field is empty (represents NULL), the field prefix contains the value -1; that is, it requires at least 1 byte. Note that if a SQL Server table column allows null values, the column requires a prefix length of 1 or greater, depending on the file storage type.

When you bulk export data and store it in either native data types or character format, use the prefix lengths

shown in the following table.

SQL SERVER

DATA TYPE char

NATIVE FORMAT

NOT NULL

varchar nchar nvarchar

float real int bigint smallint tinyint money smallmoney bit

text*

ntext*

binary varbinary

image*

datetime smalldatetime decimal numeric uniqueidentifier timestamp

NATIVE FORMAT

NULL

CHARACTER FORMAT

NULL

CHARACTER FORMAT

NOT NULL

SQL SERVER

DATA TYPE varchar(max) varbinary(max)

UDT (a user-defined data type)

XML sql_variant

NATIVE FORMAT

NOT NULL

NATIVE FORMAT

NULL

CHARACTER FORMAT

NOT NULL

CHARACTER FORMAT

NULL

*The ntext, text, and image data types will be removed in a future version of SQL Server. Avoid using these data types in new development work, and plan to modify applications that currently use them. Use nvarchar(max),

varchar(max), and varbinary(max) instead.

Prefix Lengths for Bulk Import

When data is bulk imported, the prefix length is the value that was specified when the data file was created originally. If the data file was not created by a bcp command, length prefix characters probably do not exist. In this instance, specify 0 for the prefix length.

To specify a prefix length in a data file that was not created by using bcp, use the lengths provided in Prefix Lengths for

Bulk Export , earlier in this topic.

Specify Field Length by Using bcp (SQL Server)

3/24/2017 • 4 min to read • Edit Online

The field length indicates the maximum number of characters that are required to represent data in character format. The field length is already known if the data is stored in the native format; for example, the int data type takes 4 bytes. If you have indicated 0 for the prefix length, the bcp command prompts you for field length, the default field lengths, and the impact of field-length on data storage in data files that contain char data.

The bcp Prompt for Field Length

Enter length of field <field_name> [<default>]:

For an example that shows this prompt in context, see Specify Data Formats for Compatibility when Using bcp

field in a non-XML format file. For more information on non-XML format files, see Non-XML Format Files (SQL Server)

After you interactively specify all of the fields in a bcp command, the command prompts you save your responses for each

Whether a bcp command prompts for field length depends on several factors, as follows:

When you copy data types that are not of fixed length and you specify a prefix length of 0, bcp prompts for a field length.

When converting noncharacter data to character data, bcp suggests a default field length large enough to store the data.

If the file storage type is noncharacter, the bcp command does not prompt for a field length. The data is stored in the Microsoft SQL Server native data representation (native format).

Using Default Field Lengths

Generally, Microsoft recommends that you accept the bcp-suggested default values for the field length. When a character mode data file is created, using the default field length ensures that data is not truncated and that numeric overflow errors do not occur.

If you specify a field length that is incorrect, problems can occur. For instance, if you copy numeric data and you specify a field length that is too short for the data, the bcp utility prints an overflow message and does not copy the data. Also, if you export datetime data and specify a field length of less than 26 bytes for the character string, the bcp utility truncates the data without an error message.

When the default size option is used, SQL Server expects to read an entire string. In some situations, use of a default field length can lead to an "unexpected end of file" error. Typically, this error occurs with the money and datetime data types when only part of the expected field occurs in the data file; for example, when a datetime value of mm/dd/yy is specified without the time component and is, therefore, shorter than the expected 24 character length of a datetime value in char format. To avoid this type of error, use field terminators or fixed-length data fields, or change the default field length by specifying another value.

bit binary varbinary image datetime smalldatetime float real int bigint smallint tinyint money

The following table lists the default field lengths for data to be stored as a character-file storage type. Nullable data is the same length as nonnull data.

DATA TYPE char varchar nchar nvarchar

Text ntext

DEFAULT LENGTH (CHARACTERS)

Length defined for the column

Twice the length defined for the column

Twice the length defined for the column + 1

DATA TYPE smallmoney decimal numeric uniqueidentifier timestamp

DEFAULT LENGTH (CHARACTERS)

41*

varchar(max) varbinary(max) nvarchar(max)

UDT

XML

Length of the user-defined term (UDT) column

*For more information about the decimal and numeric data types, see decimal and numeric (Transact-SQL) .

A column of type tinyint can have values from 0 through 255; the maximum number of characters that are needed to represent any number in that range is three (representing values 100 through 255).

The following table lists the default field lengths for data to be stored as native file storage type. Nullable data is the same length as nonnull data, and character data is always stored in character format.

DATA TYPE DEFAULT LENGTH (CHARACTERS) bit binary varbinary image datetime smalldatetime float real

Length defined for the column

int

DATA TYPE bigint smallint tinyint money smallmoney

DEFAULT LENGTH (CHARACTERS)

decimal numeric uniqueidentifier timestamp

*For more information about the decimal and numeric data types, see decimal and numeric (Transact-SQL) .

In all of the preceding cases, to create a data file for later reloading into SQL Server that keeps the storage space to a minimum, use a length prefix with the default file storage type and the default field length.

Specify Field and Row Terminators (SQL Server)

3/24/2017 • 7 min to read • Edit Online

For character data fields, optional terminating characters allow you to mark the end of each field in a data file with a field terminator and the end of each row with a row terminator. Terminating characters are one way to indicate to programs that read the data file where one field or row ends and another field or row begins.

When you use native or Unicode native format, use length prefixes rather than field terminators. Native format data can conflict with terminators because a native-format data file is stored in the Microsoft SQL Server internal binary data format.

Characters Supported As Terminators

The bcp command, BULK INSERT statement, and the OPENROWSET bulk rowset provider support a variety of characters as field or row terminators and always look for the first instance of each terminator. The following table lists the supported characters for terminators.

TERMINATING CHARACTER INDICATED BY

Tab

Newline character

This is the default field terminator.

This is the default row terminator.

Carriage return/line feed

Backslash*

Null terminator (nonvisible terminator)** \0

Any printable character (control characters are not printable, except null, tab, newline, and carriage return)

(*, A, t, l, and so on)

String of up to 10 printable characters, including some or all of the terminators listed earlier

(*\t\*, end, !!!!!!!!!!, \t—\n, and so on)

*Only the t, n, r, 0 and '\0' characters work with the backslash escape character to produce a control character.

**Even though the null control character (\0) is not visible when printed, it is a distinct character in the data file.

This means that using the null control character as a field or row terminator is different than having no field or row terminator at all.

If a terminator character occurs within the data, it is interpreted as a terminator, not as data, and the data after that character is interpreted as belonging to the next field or record. Therefore, choose your terminators carefully to make sure that they never appear in your data. For example, a low surrogate field terminator would not be a good choice for a field terminator if the data contains that low surrogate.

Using Row Terminators

The row terminator can be the same character as the terminator for the last field. Generally, however, a distinct row terminator is useful. For example, to produce tabular output, terminate the last field in each row with the newline character (\n) and all other fields with the tab character (\t). To place each data record on its own line in the data file, specify the combination \r\n as the row terminator.

When you use bcp interactively and specify \n (newline) as the row terminator, bcp automatically prefixes it with a \r

(carriage return) character, which results in a row terminator of \r\n.

Specifying Terminators for Bulk Export

When you bulk export char or nchar data, and want to use a non-default terminator, you must specify the terminator to the bcp command. You can specify terminators in any of the following ways:

With a format file that specifies the terminator on a field-by-field basis.

For information about how to use format files, see Format Files for Importing or Exporting Data (SQL Server)

Without a format file, the following alternatives exist:

Using the -t switch to specify the field terminator for all the fields except the last field in the row and using the -r switch to specify a row terminator.

Using a character-format switch (-c or -w) without the -t switch, which sets the field terminator to the tab character, \t. This is the same as specifying -t\t.

If you specify the -n (native data) or -N (Unicode native) switch, terminators are not inserted.

If an interactive bcp command contains the in or out option without either the format file switch

(-f) or a data-format switch (-n, -c, -w, or -N), and you have chosen not to specify prefix length and field length, the command prompts for the field terminator of each field, with a default of none:

Enter field terminator [none]:

Generally, the default is a suitable choice. However, for char or nchar data fields, see the following subsection, "Guidelines for Using Terminators." For an example that shows this prompt

in context, see Specify Data Formats for Compatibility when Using bcp (SQL Server)

After you interactively specify all of the fields in a bcp command, the command prompts you save your responses for each field in a non-XML format file. For more information about non-XML format files, see

Non-XML Format Files (SQL Server)

In some situations, a terminator is useful for a char or nchar data field. For example:

For a data column that contains a null value in a data file that will be imported into a program that does not understand the prefix length information.

Any data column that contains a null value is considered variable length. In the absence of prefix lengths, a terminator is necessary to identify the end of a null field, making sure that the data is correctly interpreted.

For a long fixed-length column whose space is only partially used by many rows.

In this situation, specifying a terminator can minimize storage space allowing the field to be treated as a variable-length field.

This example bulk exports the data from the

AdventureWorks.HumanResources.Department

table to the

Department-c-t.txt

data file using character format, with a comma as a field terminator and the newline character (\n) as the row terminator.

The bcp command contains the following switches.

SWITCH

-c

-t

DESCRIPTION

Specifies that the data fields be loaded as character data.

Specifies a comma (,) as the field terminator.

-r \n

-T

Specifies the row terminator as a newline character. This is the default row terminator, so specifying it is optional.

Specifies that the bcp utility connects to SQL Server with a trusted connection using integrated security. If -T is not specified, you need to specify -U and -P to successfully log in.

For more information, see bcp Utility .

At the Microsoft Windows command prompt enter: bcp AdventureWorks.HumanResources.Department out C:\myDepartment-c-t.txt -c -t, -r \n -T

This creates

Department-c-t.txt

, which contains 16 records with four fields each. The fields are separated by a comma.

Specifying Terminators for Bulk Import

When you bulk import char or nchar data, the bulk-import command must recognize the terminators that are used in the data file. How terminators can be specified depends on the bulk-import command, as follows:

bcp

Specifying terminators for an import operation uses the same syntax as for an export operation. For more information, see "Specifying Terminators for Bulk Export," earlier in this topic.

BULK INSERT

Terminators can be specified for individual fields in a format file or for the whole data file by using the qualifiers shown in the following table.

QUALIFIER

FIELDTERMINATOR ='field_terminator'

DESCRIPTION

Specifies the field terminator to be used for character and Unicode character data files.

The default is \t (tab character).

ROWTERMINATOR ='row_terminator' Specifies the row terminator to be used for character and Unicode character data files.

The default is \n (newline character).

For more information, see BULK INSERT (Transact-SQL) .

INSERT ... SELECT * FROM OPENROWSET(BULK...)

For the OPENROWSET bulk rowset provider, terminators can be specified only in the format file (which is required except for large-object data types). If a character data file uses a non-default terminator, it must

be defined in the format file. For more information, see Create a Format File (SQL Server)

and Use a

Format File to Bulk Import Data (SQL Server) .

For more information about the OPENROWSET BULK clause, see OPENROWSET (Transact-SQL) .

The examples in this section bulk import character data form the

Department-c-t.txt

data file created in the preceding example into the myDepartment

table in the AdventureWorks2012 sample database. Before you can run the examples, you must create this table. To create this table under the dbo schema, in SQL Server

Management Studio Query Editor, execute the following code:

USE AdventureWorks;

DROP TABLE myDepartment;

CREATE TABLE myDepartment

(DepartmentID smallint,

Name nvarchar(50),

GroupName nvarchar(50) NULL,

ModifiedDate datetime not NULL CONSTRAINT DF_AddressType_ModifiedDate DEFAULT (GETDATE())

);

The following example bulk imports the

Department-c-t.txt

data file using a bcp

command. This command uses the same command switches as the bulk export command. For more information, see "Specifying

Terminators for Bulk Export," earlier in this topic.

At the Windows command prompt enter:

bcp AdventureWorks..myDepartment in C:\myDepartment-c-t.txt -c -t , -r \n -T

The following example bulk imports the

Department-c-t.txt

data file using a

BULK INSERT

statement that uses the qualifiers shown in the following table.

OPTION

DATAFILETYPE ='char'

ATTRIBUTE

Specifies that the data fields be loaded as character data.

FIELDTERMINATOR ='

Specifies a comma (

) as the field terminator.

ROWTERMINATOR ='

Specifies the row terminator as a newline character.

In SQL Server Management Studio Query Editor, execute the following code:

USE AdventureWorks;

BULK INSERT myDepartment FROM 'C:\myDepartment-c-t.txt'

WITH (

DATAFILETYPE = 'char',

FIELDTERMINATOR = ',',

ROWTERMINATOR = '\n'

);

Keep Nulls or Use Default Values During Bulk Import

(SQL Server)

3/24/2017 • 7 min to read • Edit Online

THIS TOPIC APPLIES TO: SQL Server (starting with 2008) Azure SQL Database Azure SQL Data

Warehouse Parallel Data Warehouse

By default, when data is imported into a table, the bcp command and BULK INSERT statement observe any defaults that are defined for the columns in the table. For example, if there is a null field in a data file, the default value for the column is loaded instead. The bcp command and BULK INSERT statement both allow you to specify that nulls values be retained.

In contrast, a regular INSERT statement retains the null value instead of inserting a default value. The INSERT ...

SELECT * FROM OPENROWSET(BULK...) statement provides the same basic behavior as regular INSERT but additionally supports a table hint for inserting the default values.

OUTLINE

Keeping Null Values

Using Default Values with INSERT ... SELECT * FROM OPENROWSET(BULK...)

Keep Nulls or Use Default Values During Bulk Import

● Using bcp and Keeping Null Values without a Format File

● Using bcp and Keeping Null Values with a Non-XML Format File

● Using bcp and Using Default Values without a Format File

● Using bcp and Using Default Values with a Non-XML Format File

● Using BULK INSERT and Keeping Null Values without a Format File

● Using BULK INSERT and Keeping Null Values with a Non-XML Format File

● Using BULK INSERT and Using Default Values without a Format File

● Using BULK INSERT and Using Default Values with a Non-XML Format File

● Using OPENROWSET(BULK...) and Keeping Null Values with a Non-XML Format File

● Using OPENROWSET(BULK...) and Using Default Values with a Non-XML Format File

Keeping Null Values

The following qualifiers specify that an empty field in the data file retains its null value during the bulk-import operation, rather than inheriting a default value (if any) for the table columns. For OPENROWSET , by default, any columns that are not specified in the bulk-load operation are set to NULL.

COMMAND

bcp

QUALIFIER

-k

QUALIFIER TYPE

Switch

BULK INSERT KEEPNULLS\* Argument

INSERT ... SELECT * FROM

OPENROWSET(BULK...)

N/A N/A

\* For BULK INSERT , if default values are not available, the table column must be defined to allow null values.

These qualifiers disable checking of DEFAULT definitions on a table by these bulk-import commands. However, for any concurrent INSERT statements, DEFAULT definitions are expected.

Using Default Values with INSERT ... SELECT * FROM

OPENROWSET(BULK...)

You can specify that for an empty field in the data file, the corresponding table column uses its default value (if any). To use default values, use the table hint KEEPDEFAULTS .

For more information, see INSERT (Transact-SQL) , SELECT (Transact-SQL) , OPENROWSET (Transact-SQL) , and Table Hints

(Transact-SQL)

Example Test Conditions

The examples in this topic are based on the table, data file, and format file defined below.

Sample Table

The script below creates a test database and a table named myNulls

. Notice that the fourth table column,

Kids

, has a default value. Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

CREATE DATABASE TestDatabase;

USE TestDatabase;

CREATE TABLE dbo.myNulls (

PersonID smallint not null,

FirstName varchar(25),

LastName varchar(30),

Kids varchar(13) DEFAULT 'Default Value',

BirthDate date

);

Using Notepad, create an empty file

D:\BCP\myNulls.bcp

and insert the data below. Note that there is no value in the third record, fourth column.

1,Anthony,Grosse,Yes,1980-02-23

2,Alica,Fatnowna,No,1963-11-14

3,Stella,Rosenhain,,1992-03-02

Alternatively, you can execute the following PowerShell script to create and populate the data file:

cls

# revise directory as desired

$dir = 'D:\BCP\';

$bcpFile = $dir + 'MyNulls.bcp';

# Confirm directory exists

IF ((Test-Path -Path $dir) -eq 0)

{

Write-Host "The path $dir does not exist; please create or modify the directory.";

RETURN;

};

# clear content, will error if file does not exist, can be ignored

Clear-Content -Path $bcpFile -ErrorAction SilentlyContinue;

# Add data

Add-Content -Path $bcpFile -Value '1,Anthony,Grosse,Yes,1980-02-23';

Add-Content -Path $bcpFile -Value '2,Alica,Fatnowna,No,1963-11-14';

Add-Content -Path $bcpFile -Value '3,Stella,Rosenhain,,1992-03-02';

#Review content

Get-Content -Path $bcpFile;

Invoke-Item $bcpFile;

SQL Server support two types of format file: non-XML format and XML format. The non-XML format is the

original format that is supported by earlier versions of SQL Server. Please review Non-XML Format Files (SQL

Server) for detailed information. The following command will use the bcp utility to generate a non-xml format file,

myNulls.fmt

, based on the schema of myNulls

. To use a bcp command to create a format file, specify the format argument and use nul instead of a data-file path. The format option also requires the -f option. In addition, for

this example, the qualifier c is used to specify character data, t, is used to specify a comma as a field terminator

, and T is used to specify a trusted connection using integrated security. At a command prompt, enter the following command: bcp TestDatabase.dbo.myNulls format nul -c -f D:\BCP\myNulls.fmt -t, -T

REM Review file

Notepad D:\BCP\myNulls.fmt

Ensure your non-XML format file ends with a carriage return\line feed. Otherwise you will likely receive the following error message:

SQLState = S1000, NativeError = 0

Error = [Microsoft][ODBC Driver 13 for SQL Server]I/O error while reading BCP format file

For more information about creating format files, see Create a Format File (SQL Server)

Keep Nulls or Use Default Values During Bulk Import

The examples below use the database, datafile, and format files created above.

Using bcp and Keeping Null Values without a Format File

-k switch. At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myNulls;"

REM Import data bcp TestDatabase.dbo.myNulls IN D:\BCP\myNulls.bcp -c -t, -T -k

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myNulls;"

Using bcp

and Keeping Null Values with a Non-XM L Format File

-k and -f switches. At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myNulls;"

REM Import data bcp TestDatabase.dbo.myNulls IN D:\BCP\myNulls.bcp -f D:\BCP\myNulls.fmt -T -k

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myNulls;"

Using bcp and Using Default Values without a Format File

At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myNulls;"

REM Import data bcp TestDatabase.dbo.myNulls IN D:\BCP\myNulls.bcp -c -t, -T

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myNulls;"

Using bcp

and Using Default Values with a Non-XM L Format File

-f switch. At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myNulls;"

REM Import data bcp TestDatabase.dbo.myNulls IN D:\BCP\myNulls.bcp -f D:\BCP\myNulls.fmt -T

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myNulls;"

Using B ULK INSERT and Keeping Null Values without a Format File

KEEPNULLS argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myNulls; -- for testing

BULK INSERT dbo.myNulls

FROM 'D:\BCP\myNulls.bcp'

WITH (

DATAFILETYPE = 'char',

FIELDTERMINATOR = ',',

KEEPNULLS

);

-- review results

SELECT * FROM TestDatabase.dbo.myNulls;

Using B ULK INSERT

and Keeping Null Values with a Non-XM L Format File

KEEPNULLS and the FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server

Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myNulls; -- for testing

BULK INSERT dbo.myNulls

FROM 'D:\BCP\myNulls.bcp'

WITH (

FORMATFILE = 'D:\BCP\myNulls.fmt',

KEEPNULLS

);

-- review results

SELECT * FROM TestDatabase.dbo.myNulls;

Using B ULK INSERT and Using Default Values without a Format File

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myNulls; -- for testing

BULK INSERT dbo.myNulls

FROM 'D:\BCP\myNulls.bcp'

WITH (

DATAFILETYPE = 'char',

FIELDTERMINATOR = ','

);

-- review results

SELECT * FROM TestDatabase.dbo.myNulls;

Using B ULK INSERT

and Using Default Values with a Non-XM L Format File

FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myNulls; -- for testing

BULK INSERT dbo.myNulls

FROM 'D:\BCP\myNulls.bcp'

WITH (

FORMATFILE = 'D:\BCP\myNulls.fmt'

);

-- review results

SELECT * FROM TestDatabase.dbo.myNulls;

Using OPENROWSET(B ULK...)

and Keeping Null Values with a Non-XM L Format File

FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myNulls; -- for testing

INSERT INTO dbo.myNulls

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myNulls.bcp',

FORMATFILE = 'D:\BCP\myNulls.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myNulls;

Using OPENROWSET(B ULK...)

and Using Default Values with a Non-XM L Format File

KEEPDEFAULTS table hint and FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL

Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myNulls; -- for testing

INSERT INTO dbo.myNulls

WITH (KEEPDEFAULTS)

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myNulls.bcp',

FORMATFILE = 'D:\BCP\myNulls.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myNulls;

Prepare Data for Bulk Export or Import (SQL Server)

To use a format file

Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

Use a Format File to Skip a Data Field (SQL Server)

Use a Format File to Skip a Table Column (SQL Server)

To use data formats for bulk import or bulk export

Import Native and Character Format Data from Earlier Versions of SQL Server

Use Character Format to Import or Export Data (SQL Server)

Use Native Format to Import or Export Data (SQL Server)

Use Unicode Character Format to Import or Export Data (SQL Server)

Use Unicode Native Format to Import or Export Data (SQL Server)

To specify data formats for compatibility when using bcp

Specify Prefix Length in Data Files by Using bcp (SQL Server)

Specify File Storage Type by Using bcp (SQL Server)

Keep Identity Values When Bulk Importing Data

(SQL Server)

3/24/2017 • 7 min to read • Edit Online

Data files that contain identity values can be bulk imported into an instance of Microsoft SQL Server. By default, the values for the identity column in the data file that is imported are ignored and SQL Server assigns unique values automatically. The unique values are based on the seed and increment values that are specified during table creation.

If the data file does not contain values for the identifier column in the table, use a format file to specify that the

identifier column in the table should be skipped when importing data. See Use a Format File to Skip a Table

Column (SQL Server) for additional information.

OUTLINE

Keep Identity Values

● Using bcp and Keeping Identity Values without a Format File

● Using bcp and Keeping Identity Values with a Non-XML Format File

● Using bcp and Generated Identity Values without a Format File

● Using bcp and Generated Identity Values with a Non-XML Format File

● Using BULK INSERT and Keeping Identity Values without a Format File

● Using BULK INSERT and Keeping Identity Values with a Non-XML Format File

● Using BULK INSERT and Generated Identity Values without a Format File

● Using BULK INSERT and Generated Identity Values with a Non-XML Format File

● Using OPENROWSET and Keeping Identity Values with a Non-XML Format File

● Using OPENROWSET and Generated Identity Values with a Non-XML Format File

Keep Identity Values

To prevent SQL Server from assigning identity values while bulk importing data rows into a table, use the appropriate keep-identity command qualifier. When you specify a keep-identity qualifier, SQL Server uses the identity values in the data file. These qualifiers are as follows:

COMMAND

bcp

BULK INSERT

INSERT ... SELECT * FROM

OPENROWSET(BULK...)

KEEP-IDENTITY QUALIFIER

-E

KEEPIDENTITY

QUALIFIER TYPE

Switch

Argument

Table hint

For more information, see bcp Utility , BULK INSERT (Transact-SQL) , OPENROWSET (Transact-SQL) , INSERT

(Transact-SQL) , SELECT (Transact-SQL) , and Table Hints (Transact-SQL) .

To create an automatically incrementing number that can be used in multiple tables or that can be called from applications without referencing any table, see Sequence Numbers .

Example Test Conditions

The examples in this topic are based on the table, data file, and format file defined below.

The script below creates a test database and a table named myIdentity

. Execute the following Transact-SQL in

Microsoft SQL Server Management Studio (SSMS):

CREATE DATABASE TestDatabase;

USE TestDatabase;

CREATE TABLE dbo.myIdentity (

PersonID smallint IDENTITY(1,1) NOT NULL,

FirstName varchar(25) NOT NULL,

LastName varchar(30) NOT NULL,

BirthDate date

);

Using Notepad, create an empty file

D:\BCP\myIdentity.bcp

and insert the data below.

3,Anthony,Grosse,1980-02-23

2,Alica,Fatnowna,1963-11-14

1,Stella,Rosenhain,1992-03-02

4,Miller,Dylan,1954-01-05

Alternatively, you can execute the following PowerShell script to create and populate the data file:

cls

# revise directory as desired

$dir = 'D:\BCP\';

$bcpFile = $dir + 'myIdentity.bcp';

# Confirm directory exists

IF ((Test-Path -Path $dir) -eq 0)

{

Write-Host "The path $dir does not exist; please create or modify the directory.";

RETURN;

};

# clear content, will error if file does not exist, can be ignored

Clear-Content -Path $bcpFile -ErrorAction SilentlyContinue;

# Add data

Add-Content -Path $bcpFile -Value '3,Anthony,Grosse,1980-02-23';

Add-Content -Path $bcpFile -Value '2,Alica,Fatnowna,1963-11-14';

Add-Content -Path $bcpFile -Value '1,Stella,Rosenhain,1992-03-02';

Add-Content -Path $bcpFile -Value '4,Miller,Dylan,1954-01-05';

#Review content

Get-Content -Path $bcpFile;

Invoke-Item $bcpFile;

SQL Server support two types of format file: non-XML format and XML format. The non-XML format is the original

format that is supported by earlier versions of SQL Server. Please review Non-XML Format Files (SQL Server)

for detailed information. The following command will use the bcp utility to generate a non-xml format file, myIdentity.fmt

, based on the schema of myIdentity

. To use a bcp command to create a format file, specify the

format argument and use nul instead of a data-file path. The format option also requires the -f option. In

addition, for this example, the qualifier c is used to specify character data, t, is used to specify a comma as a field terminator , and T is used to specify a trusted connection using integrated security. At a command prompt, enter

the following command: bcp TestDatabase.dbo.myIdentity format nul -c -f D:\BCP\myIdentity.fmt -t, -T

REM Review file

Notepad D:\BCP\myIdentity.fmt

Ensure your non-XML format file ends with a carriage return\line feed. Otherwise you will likely receive the following error message:

SQLState = S1000, NativeError = 0

Error = [Microsoft][ODBC Driver 13 for SQL Server]I/O error while reading BCP format file

Examples

The examples below use the database, datafile, and format files created above.

Using bcp and Keeping Identity Values without a Format File

-E switch. At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myIdentity;"

REM Import data bcp TestDatabase.dbo.myIdentity IN D:\BCP\myIdentity.bcp -T -c -t, -E

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myIdentity;"

Using bcp

and Keeping Identity Values with a Non-XM L Format File

-E and -f switches. At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myIdentity;"

REM Import data bcp TestDatabase.dbo.myIdentity IN D:\BCP\myIdentity.bcp -f D:\BCP\myIdentity.fmt -T -E

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myIdentity;"

Using bcp and Generated Identity Values without a Format File

Using defaults. At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myIdentity;"

REM Import data bcp TestDatabase.dbo.myIdentity IN D:\BCP\myIdentity.bcp -T -c -t,

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myIdentity;"

Using bcp

and Generated Identity Values with a Non-XM L Format File

Using defaults and -f switch. At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.myIdentity;"

REM Import data bcp TestDatabase.dbo.myIdentity IN D:\BCP\myIdentity.bcp -f D:\BCP\myIdentity.fmt -T

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.myIdentity;"

Using B ULK INSERT and Keeping Identity Values without a Format File

KEEPIDENTITY argument. Execute the following Transact-SQL in Microsoft SQL Server Management Studio

(SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myIdentity; -- for testing

BULK INSERT dbo.myIdentity

FROM 'D:\BCP\myIdentity.bcp'

WITH (

DATAFILETYPE = 'char',

FIELDTERMINATOR = ',',

KEEPIDENTITY

);

-- review results

SELECT * FROM TestDatabase.dbo.myIdentity;

Using B ULK INSERT

and Keeping Identity Values with a Non-XM L Format File

KEEPIDENTITY and the FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server

Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myIdentity; -- for testing

BULK INSERT dbo.myIdentity

FROM 'D:\BCP\myIdentity.bcp'

WITH (

FORMATFILE = 'D:\BCP\myIdentity.fmt',

KEEPIDENTITY

);

-- review results

SELECT * FROM TestDatabase.dbo.myIdentity;

Using B ULK INSERT and Generated Identity Values without a Format File

Using defaults. Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myIdentity; -- for testing

BULK INSERT dbo.myIdentity

FROM 'D:\BCP\myIdentity.bcp'

WITH (

DATAFILETYPE = 'char',

FIELDTERMINATOR = ','

);

-- review results

SELECT * FROM TestDatabase.dbo.myIdentity;

Using B ULK INSERT

and Generated Identity Values with a Non-XM L Format File

Using defaults and FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server

Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myIdentity; -- for testing

BULK INSERT dbo.myIdentity

FROM 'D:\BCP\myIdentity.bcp'

WITH (

FORMATFILE = 'D:\BCP\myIdentity.fmt'

);

-- review results

SELECT * FROM TestDatabase.dbo.myIdentity;

Using OPENROWSET(B ULK...)

and Keeping Identity Values with a Non-XM L Format File

KEEPIDENTITY table hint and FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL

Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myIdentity; -- for testing

INSERT INTO dbo.myIdentity

WITH (KEEPIDENTITY)

(PersonID, FirstName, LastName, BirthDate)

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myIdentity.bcp',

FORMATFILE = 'D:\BCP\myIdentity.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myIdentity;

Using OPENROWSET(B ULK...)

and Generated Identity Values with a Non-XM L Format File

Using defaults and FORMATFILE argument. Execute the following Transact-SQL in Microsoft SQL Server

Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE dbo.myIdentity; -- for testing

INSERT INTO dbo.myIdentity

(FirstName, LastName, BirthDate)

SELECT FirstName, LastName, BirthDate

FROM OPENROWSET (

BULK 'D:\BCP\myIdentity.bcp',

FORMATFILE = 'D:\BCP\myIdentity.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myIdentity;

Prepare Data for Bulk Export or Import (SQL Server)

To use a format file

Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

Use a Format File to Skip a Data Field (SQL Server)

Use a Format File to Skip a Table Column (SQL Server)

To use data formats for bulk import or bulk export

Import Native and Character Format Data from Earlier Versions of SQL Server

Use Character Format to Import or Export Data (SQL Server)

Use Native Format to Import or Export Data (SQL Server)

Use Unicode Character Format to Import or Export Data (SQL Server)

Use Unicode Native Format to Import or Export Data (SQL Server)

To specify data formats for compatibility when using bcp

Specify Prefix Length in Data Files by Using bcp (SQL Server)

Specify File Storage Type by Using bcp (SQL Server)

Bulk Import Large-Object Data with OPENROWSET

Bulk Rowset Provider

3/29/2017 • 1 min to read • Edit Online

THIS TOPIC APPLIES TO: SQL Server (starting with 2016) Azure SQL Database Azure SQL Data

Warehouse Parallel Data Warehouse

The SQL Server OPENROWSET Bulk Rowset Provider enables you to bulk import a data file as large-object data.

The large-object data types supported by OPENROWSET Bulk Rowset Provider are varbinary(max) or image,

varchar(max) or text, and nvarchar(max) or ntext.

The image, text and ntext data types are deprecated.

The OPENROWSET BULK clause supports three options for importing the contents of a data file as a single-row, single-column rowset. You can specify one of these large-object options instead of using a format file. These options are as follows:

SINGLE_BLOB

Reads the contents of data_file as a single-row, returns the contents as a single-column rowset of type

varbinary(max).

SINGLE_CLOB

Reads the contents of the specified data file as characters, returns the contents as a single-row, single-column rowset of type varchar(max), using the collation of the current database; such as a text or Microsoft Word document.

SINGLE_NCLOB

Reads the contents of the specified data file as Unicode, returns the contents as a single-row, single-column rowset of type nvarchar(max), using the collation of the current database.

Format Files for Importing or Exporting Data (SQL

Server)

3/24/2017 • 3 min to read • Edit Online

When you bulk import data into a SQL Server table or bulk export data from a table, you can use a format file to store all the format information that is required to bulk export or bulk import data. This includes format information for each field in a data file relative to that table.

SQL Server 2017 supports two types of format files: XML formats and non-XML format files. Both non-XML format files and XML format files contain descriptions of every field in a data file, and XML format files also contain descriptions of the corresponding table columns. Generally, XML and non-XML format files are interchangeable. However, we recommend that you use the XML syntax for new format files because they

provide several advantages over non-XML format files. For more information, see XML Format Files (SQL

HumanResources.myTeam Sample Table (SQL Server)

Benefits of Format Files

Provides a flexible system for writing data files that requires little or no editing to comply with other data formats or to read data files from other software.

Enables you to bulk import data without having to add or delete unnecessary data or to reorder existing data in the data file. Format files are particularly useful when a mismatch exists between fields in the data file and columns in the table.

Examples of Format Files

The following examples show the layout of a non-XML format file and of an XML format file. These format files correspond to the

HumanResources.myTeam

table in the AdventureWorks2012 sample database. This table contains four columns:

EmployeeID

Name

Title

, and

ModifiedDate

For information about this table and how to create it, see

The following non-XML format file uses the SQL Server native data format for the

HumanResources.myTeam

table.

This format file was created by using the following bcp

command.

bcp AdventureWorks.HumanResources.myTeam format nul -f myTeam.Fmt -n -T

The contents of this format file are as follows: 9.0

1 SQLSMALLINT 0 2 "" 1 EmployeeID ""

2 SQLNCHAR 2 100 "" 2 Name SQL_Latin1_General_CP1_CI_AS

3 SQLNCHAR 2 100 "" 3 Title SQL_Latin1_General_CP1_CI_AS

4 SQLNCHAR 2 100 "" 4 Background SQL_Latin1_General_CP1_CI_AS

For more information, see Non-XML Format Files (SQL Server)

The following XML format file uses the SQL Server native data format for the

HumanResources.myTeam

table. This

format file was created by using the following bcp

command.

bcp AdventureWorks.HumanResources.myTeam format nul -f myTeam.Xml -x -n -T

The format file contains:

<?xml version="1.0"?>

<FIELD ID="2" xsi:type="NCharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

<FIELD ID="3" xsi:type="NCharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

<FIELD ID="4" xsi:type="NCharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

</ROW>

</BCPFORMAT>

For more information, see XML Format Files (SQL Server)

When Is a Format File Required?

An INSERT ... SELECT * FROM OPENROWSET(BULK...) statement always requires a format file.

For bcp or BULK INSERT, in simple situations, using a format file is optional and rarely necessary.

However, for complex bulk-import situations, a format file is frequently required.

Format files are required if:

The same data file is used as a source for multiple tables that have different schemas.

The data file has a different number of fields that the target table has columns; for example:

The target table contains at least one column for which either a default value is defined or NULL is allowed.

The users do not have SELECT/INSERT permissions on one or more columns in the table.

A single data file is used with two or more tables that have different schemas.

The column order is different for the data file and table.

The terminating characters or prefix lengths differ among the columns of the data file.

In the absence of a format file, if a bcp command specifies a data-format switch (-n, -c, -w, or -N) or a BULK INSERT operation specifies the DATAFILETYPE option, the specified data format is used as the default method of interpreting the fields of the data file.

Related Tasks

Use a Format File to Skip a Table Column (SQL Server)

Use a Format File to Skip a Data Field (SQL Server)

Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

Non-XML Format Files (SQL Server)

3/24/2017 • 5 min to read • Edit Online

In SQL Server 2017, two types of format files are supported for bulk exporting and importing: non-XML

format files and XML format files.

In this Topic:

Benefits

Structure of Non-XML Format Files

Example of a Non-XML Format File

field. For more information, see Specify Data Formats for Compatibility when Using bcp (SQL

Benefits of Non-XML Format Files

You can create a non-XML format file automatically by specifying the format option in a bcp command.

When you specify an existing format file in a bcp command, the command uses the values that are recorded in the format file and does not prompt you for the file storage type, prefix length, field length, or field terminator.

You can create a format file for a particular data type such as character data or native data.

You can create a non-XML format file that contains interactively specified attributes for each data

XML format files offer several advantages over non-XML format files. For more information, see XML Format Files

information, see Specify File Storage Type by Using bcp

Structure of Non-XML Format Files

A non-XML format file is a text file that has a specific structure. The non-XML format file contains information about the file storage type, prefix length, field length, and field terminator of every table column.

The following illustration illustrates the format-file fields for a sample non-XML format file.

The Version and Number of columns fields occur one time only. Their meanings are describes in the following table.

FORMAT-FILE FIELD

Version

DESCRIPTION

Version number of the bcp utility:

9.0 = SQL Server 2005

10.0 = SQL Server 2008

11.0 = SQL Server 2012

12.0 = SQL Server 2014

The version number is recognized only by bcp, not by

Transact-SQL.

Number of columns

Note: The version of the bcp utility (Bcp.exe) used to read a format file must be the same as, or a later version than was used to create the format file. For example, SQL

Server 2012bcp can read a version 10.0 format file, which is generated by SQL Server 2008bcp, but SQL

Server 2008bcp cannot read a version 12.0 format file, which is generated by SQL Server 2014bcp.

Number of fields in the data file. This number must be the same in all rows.

The other format-file fields describe the data fields that are to be bulk imported or exported. Each data field requires a separate row in the format file. Every format-file row contains values for the format-file fields that are described in the following table.

FORMAT-FILE FIELD

Host file field order

Host file data type

DESCRIPTION

A number that indicates the position of each field in the data file. The first field in the row is 1, and so on.

Indicates the data type that is stored in a given field of the data file. With ASCII data files, use SQLCHAR; for native format data files, use default data types. For more

information, see Specify Prefix Length in Data Files by

FORMAT-FILE FIELD

Prefix length

Host file data length

Terminator

Server column order

Server column name

Column collation

DESCRIPTION

Number of length prefix characters for the field. Valid prefix lengths are 0, 1, 2, 4, and 8. To avoid specifying the length prefix, set this to 0. A length prefix must be specified if the field contains NULL data values. For more

Using bcp (SQL Server) .

Maximum length, in bytes, of the data type stored in the particular field of the data file.

If you are creating a non-XML format file for a delimited text file, you can specify 0 for the host file data length of every data field. When a delimited text file having a prefix length of 0 and a terminator is imported, the field-length value is ignored, because the storage space used by the field equals the length of the data plus the terminator.

For more information, see Specify Field Length by Using bcp (SQL Server) .

Delimiter to separate the fields in a data file. Common terminators are comma (,), tab (\t), and end of line (\r\n).

For more information, see Specify Field and Row

Terminators (SQL Server) .

Order in which columns appear in the SQL Server table.

For example, if the fourth field in the data file maps to the sixth column in a SQL Server table, the server column order for the fourth field is 6.

To prevent a column in the table from receiving any data from the data file, set the server column order value to 0.

Name of the column copied from the SQL Server table.

The actual name of the field is not required, but the field in the format file must not be blank.

The collation used to store character and Unicode data in the data file.

You can modify a format file to let you bulk import from a data file in which the number or order of the fields are different from the number or order of table columns. For more information, see the

For an illustration that shows the format-file fields in relation to this sample non-XML format file, see Structure of

list, later in this topic.

Example of a Non-XML Format File

The following example shows a previously created non-XML format file ( myDepartmentIdentical-f-c.fmt

This file describes a character-data field for every column in the

HumanResources.Department

table in the

AdventureWorks2012

sample database.

The generated format file, myDepartmentIdentical-f-c.fmt

, contains the following information:

12.0

1 SQLCHAR 0 7 "\t" 1 DepartmentID ""

2 SQLCHAR 0 100 "\t" 2 Name SQL_Latin1_General_CP1_CI_AS

3 SQLCHAR 0 100 "\t" 3 GroupName SQL_Latin1_General_CP1_CI_AS

4 SQLCHAR 0 24 "\r\n" 4 ModifiedDate ""

Non-XML Format Files , earlier in this topic.

Related Tasks

Use a Format File to Skip a Table Column (SQL Server)

Use a Format File to Skip a Data Field (SQL Server)

Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

XML Format Files (SQL Server)

3/24/2017 • 19 min to read • Edit Online

SQL Server 2017 provides an XML schema that defines syntax for writing XML format files to use for bulk importing data into a SQL Server table. XML format files must adhere to this schema, which is defined in the

XML Schema Definition Language (XSDL). XML format files are only supported when SQL Server tools are installed together with SQL Server Native Client.

You can use an XML format file with a bcp command, BULK INSERT statement, or INSERT ... SELECT * FROM

OPENROWSET(BULK...) statement. The bcp command allows you to automatically generate an XML format file for a table; for more information, see bcp Utility .

Two types of format files are supported for bulk exporting and importing: non-XML format files and XML format files.

XML format files provide a flexible and powerful alternative to non-XML format files. For information about non-XML format files, see

Non-XML Format Files (SQL Server)

In This Topic:

Benefits of XML Format Files

Structure of XML Format Files

Schema Syntax for XML Format Files

Sample XML Format Files

To see how the syntax corresponds to actual XML format files, see Sample XML Format Files

Benefits of XML Format Files

XML format files are self-describing, making them easy to read, create, and extend. They are human readable, making it easy to understand how data is interpreted during bulk operations.

XML format files contain the data types of target columns. The XML encoding clearly describes the data types and data elements of the data file and also the mapping between data elements and table columns.

This enables separation between how data is represented in the data file and what data type is associated with each field in the file. For example, if a data file contains a character representation of the data, the corresponding SQL column type is lost.

An XML format file allows for loading of a field that contains a single large object (LOB) data type from a data file.

An XML format file can be enhanced yet remain compatible with its earlier versions. Furthermore, the clarity of XML encoding facilitates the creation of multiple format files for a given data file. This is useful if you have to map all or some of the data fields to columns in different tables or views.

The XML syntax is independent of the direction of the operation; that is, the syntax is the same for bulk export and bulk import.

You can use XML format files to bulk import data into tables or non-partitioned views and to bulk export data.

For the OPENROWSET(BULK...) function specifying a target table is optional. This is because the function relies on the XML format file to read data from a data file.

A target table is necessary with the bcp command and the BULK INSERT statement, which uses the target table columns to do the type conversion.

Structure of XML Format Files

Like a non-XML format file, an XML format file defines the format and structure of the data fields in a data file and maps those data fields to columns in a single target table.

An XML format file possesses two main components, <RECORD> and <ROW>:

<RECORD> describes the data as it is stored in the data file.

Each <RECORD> element contains a set of one or more <FIELD> elements. These elements correspond to fields in the data file. The basic syntax is as follows:

<FIELD .../> [ ...n ]

</RECORD>

Each <FIELD> element describes the contents of a specific data field. A field can only be mapped to one column in the table. Not all fields need to be mapped to columns.

A field in a data file can be either of fixed/variable length or character terminated. A field value can be represented as: a character (using single-byte representation), a wide character (using Unicode two-byte representation), native database format, or a file name. If a field value is represented as a file name, the file name points to the file that contains the value of a BLOB column in the target table.

<ROW> describes how to construct data rows from a data file when the data from the file is imported into a SQL Server table.

A <ROW> element contains a set of <COLUMN> elements. These elements correspond to table columns. The basic syntax is as follows:

<ROW>

<COLUMN .../> [ ...n ]

</ROW>

Each <COLUMN> element can be mapped to only one field in the data file. The order of the

<COLUMN> elements in the <ROW> element defines the order in which they are returned by the bulk operation. The XML format file assigns each <COLUMN> element a local name that has no relationship to the column in the target table of a bulk import operation.

Schema Syntax for XML Format Files

This section contains a summary of the elements and attributes of the XML schema for XML format files. The syntax of a format file is independent of the direction of the operation; that is, the syntax is the same for bulk export and bulk import. This section also considers how bulk import uses the <ROW> and <COLUMN>

elements and how to put the xsi:type value of an element into a data set.

, later in this topic.

You can modify a format file to let you bulk import from a data file in which the number and/or order of the fields differ

from the number and/or order of table columns. For more information, see Format Files for Importing or Exporting Data

Basic Syntax of the XML Schema

In This Section:

How Bulk Import Uses the <ROW> Element

How Bulk Import Uses the <COLUMN> Element

Putting the xsi:type Value into a Data Set

This syntax statements show only the elements (<BCPFORMAT>, <RECORD>, <FIELD>, <ROW>, and

<COLUMN>) and their basic attributes.

<FIELD ID = "fieldID" xsi:type = "fieldType" [...]

</RECORD>

<ROW>

<COLUMN SOURCE = "fieldID" NAME = "columnName" xsi:type = "columnType" [...]

</ROW>

</BCPFORMAT>

Additional attributes that are associated with the value of the xsi:type in a <FIELD> or <COLUMN> element are described later in this topic.

In This Section:

Schema Elements

Attributes of the <FIELD> Element

(and Xsi:type values of the <FIELD> Element

)

Attributes of the <COLUMN> Element

and ( Xsi:type values of the <COLUMN> Element

)

This section summarizes the purpose of each element that the XML schema defines for XML format files. The attributes are described in separate sections later in this topic.

Is the format-file element that defines the record structure of a given data file and its correspondence to the columns of a table row in the table.

Defines a complex element containing one or more <FIELD> elements. The order in which the fields are declared in the format file is the order in which those fields appear in the data file.

Defines a field in data file, which contains data.

The attributes of this element are discussed in Attributes of the <FIELD> Element

, later in this topic.

Defines a complex element containing one or more <COLUMN> elements. The order of the <COLUMN> elements is independent of the order of <FIELD> elements in a RECORD definition. Rather, the order of the

<COLUMN> elements in a format file determines the column order of the resultant rowset. Data fields are loaded in the order in which the corresponding <COLUMN> elements are declared in the <COLUMN> element.

For more information, see How Bulk Import Uses the <ROW> Element

, later in this topic.

Defines a column as an element (<COLUMN>). Each <COLUMN> element corresponds to a <FIELD> element

(whose ID is specified in the SOURCE attribute of the <COLUMN> element).

The attributes of this element are discussed in Attributes of the <COLUMN> Element

, later in this topic. Also

see, How Bulk Import Uses the <COLUMN> Element

, later in this topic.

</BCPFORMAT>

Required to end the format file.

This section describes the attributes of the <FIELD> element, which are summarized in the following schema syntax:

<FIELD

ID ="fieldID" xsi:type ="fieldType"

[ LENGTH ="n" ]

[ PREFIX_LENGTH ="p" ]

[ MAX_LENGTH ="m" ]

[ COLLATION ="collationName" ]

[ TERMINATOR ="terminator" ]

Each <FIELD> element is independent of the others. A field is described in terms of the following attributes:

FIELD ATTRIBUTE DESCRIPTION

OPTIONAL /

REQUIRED

FIELD ATTRIBUTE

ID ="fieldID" xsi:type ="fieldType"

LENGTH ="n"

PREFIX_LENGTH ="p"

MAX_LENGTH ="m"

COLLATION ="collationName"

TERMINATOR = "terminator"

DESCRIPTION

Specifies the logical name of the field in the data file. The ID of a field is the key used to refer to the field.

<FIELD ID="fieldID"/> maps to

OPTIONAL /

REQUIRED

Required

This is an XML construct (used like an attribute) that identifies the type of the instance of the element. The value of fieldType determines which of the optional attributes (below) you need in a given instance.

Required (depending on the data type)

This attribute defines the length for an instance of a fixed-length data type.

The value of n must be a positive integer.

Optional unless required by the xsi:type value

This attribute defines the prefix length for a binary data representation. The

PREFIX_LENGTH value, p, must be one of the following: 1, 2, 4, or 8.

Optional unless required by the xsi:type value

This attribute is the maximum number of bytes that can be stored in a given field. Without a target table, the column max-length is not known. The

MAX_LENGTH attribute restricts the maximum length of an output character column, limiting the storage allocated for the column value. This is especially convenient when using the

OPENROWSET function's BULK option in a SELECT FROM clause.

The value of m must be a positive integer. By default, the maximum length is 8000 characters for a char column and 4000 characters for an

nchar column.

Optional

COLLATION is only allowed for character fields. For a list of the SQL collation names, see SQL Server

Collation Name (Transact-SQL) .

Optional

This attribute specifies the terminator of a data field. The terminator can be any character. The terminator must be a unique character that is not part of the data.

By default, the field terminator is the tab character (represented as \t). To represent a paragraph mark, use \r\n.

Used only with an xsi:type of character data, which requires this attribute

The xsi:type value is an XML construct (used like an attribute) that identifies the data type of an instance of an element. For information on using the "Putting the xsi:type Value into a Data Set," later in this section.

The xsi:type value of the <FIELD> element supports the following data types.

<FIELD> XSI:TYPE VALUES

NativeFixed

REQUIRED XML ATTRIBUTE(S)

FOR DATA TYPE

LENGTH

OPTIONAL XML ATTRIBUTE(S)

FOR DATA TYPE

None.

NativePrefix

CharFixed

NCharFixed

CharPrefix

PREFIX_LENGTH

LENGTH

PREFIX_LENGTH

MAX_LENGTH

COLLATION

MAX_LENGTH, COLLATION

NCharPrefix

CharTerm

NCharTerm

PREFIX_LENGTH

TERMINATOR

MAX_LENGTH, COLLATION

For more information about Microsoft SQL Server data types, see Data Types (Transact-SQL) .

This section describes the attributes of the <COLUMN> element, which are summarized in the following schema syntax:

<COLUMN

SOURCE = "fieldID"

NAME = "columnName" xsi:type = "columnType"

[ LENGTH = "n" ]

[ PRECISION = "n" ]

[ SCALE = "value" ]

[ NULLABLE = { "YES"

"NO" } ]

A field is mapped to a column in the target table using the following attributes:

COLUMN ATTRIBUTE DESCRIPTION

OPTIONAL /

REQUIRED

COLUMN ATTRIBUTE

SOURCE ="fieldID"

NAME = "columnName" xsi:type ="ColumnType"

LENGTH ="n"

PRECISION ="n"

SCALE ="int"

DESCRIPTION

Specifies the ID of the field being mapped to the column.

OPTIONAL /

REQUIRED

Required

Specifies the name of the column in the row set represented by the format file. This column name is used to identify the column in the result set, and it need not correspond to the column name used in the target table.

Required

This is an XML construct (used like an attribute) that identifies the data type of the instance of the element. The value of ColumnType determines which of the optional attributes

(below) you need in a given instance.

Note: The possible values of

ColumnType and their associated attributes are listed in the

<COLUMN> element table in the

Xsi:type values of the <COLUMN>

Element section.

Optional

Defines the length for an instance of a fixed-length data type. LENGTH is used only when the xsi:type is a string data type.

The value of n must be a positive integer.

Optional (available only if the xsi:type is a string data type)

Indicates the number of digits in a number. For example, the number

123.45 has a precision of 5.

The value must be a positive integer.

Optional (available only if the xsi:type is a variable-number data type)

Indicates the number of digits to the right of the decimal point in a number.

For example, the number 123.45 has a scale of 2.

The value must be an integer.

Optional (available only if the xsi:type is a variable-number data type)

COLUMN ATTRIBUTE

NULLABLE = { "YES"

"NO" }

DESCRIPTION

OPTIONAL /

REQUIRED

Indicates whether a column can assume NULL values. This attribute is completely independent of FIELDS.

However, if a column is not NULLABLE and field specifies NULL (by not specifying any value), a run-time error results.

The NULLABLE attribute is used only if you do a plain SELECT FROM

OPENROWSET(BULK...) statement.

Optional (available for any data type)

The <COLUMN> element supports native SQL data types, as follows:

TYPE CATEGORY

Fixed

REQUIRED XML ATTRIBUTE(S)

FOR DATA TYPE

OPTIONAL XML ATTRIBUTE(S)

FOR DATA TYPE

None.

NULLABLE

<COLUMN> DATA TYPES

SQLBIT, SQLTINYINT,

SQLSMALLINT, SQLINT,

SQLBIGINT, SQLFLT4,

SQLFLT8, SQLDATETIME,

SQLDATETIM4,

SQLDATETIM8,

SQLMONEY,

SQLMONEY4,

SQLVARIANT, and

SQLUNIQUEID

Variable Number SQLDECIMAL and

SQLNUMERIC

None.

NULLABLE, PRECISION,

SCALE

LOB

Character LOB

Binary string

Character string

SQLIMAGE, CharLOB,

SQLTEXT, and SQLUDT

SQLNTEXT

SQLBINARY and

SQLVARYBIN

SQLCHAR,

SQLVARYCHAR,

SQLNCHAR, and

SQLNVARCHAR

None.

NULLABLE

NULLABLE, LENGTH

To bulk export or import SQLXML data, use one of the following data types in your format file: SQLCHAR or

SQLVARYCHAR (the data is sent in the client code page or in the code page implied by the collation), SQLNCHAR or

SQLNVARCHAR (the data is sent as Unicode), or SQLBINARY or SQLVARYBIN (the data is sent without any conversion).

For more information about SQL Server data types, see Data Types (Transact-SQL) .

The <ROW> element is ignored in some contexts. Whether the <ROW> element affects a bulk-import operation depends on how the operation is performed: the bcp command

When data is loaded into a target table, bcp ignores the <ROW> component. Instead, bcp loads the data based on the column types of the target table.

Transact-SQL statements (BULK INSERT and OPENROWSET's Bulk rowset provider)

When bulk importing data into a table, Transact-SQL statements use the <ROW> component to generate the input rowset. Also, Transact-SQL statements perform appropriate type conversions based on the column types specified under <ROW> and the corresponding column in the target table. If a mismatch exists between column types as specified in the format file and in the target table, an extra type conversion occurs. This extra type conversion may lead to some discrepancy (that is, a loss of precision) in behavior in BULK INSERT or OPENROWSET's Bulk rowset provider as compared to bcp.

The information in the <ROW> element allows a row to be constructed without requiring any additional information. For this reason, you can generate a rowset using a SELECT statement (SELECT * FROM

OPENROWSET(BULK datafile FORMATFILE=xmlformatfile).

The OPENROWSET BULK clause requires a format file (note that converting from the data type of the field to the data type of a column is available only with an XML format file).

For bulk importing data into a table, the <COLUMN> elements in a format file map a data-file field to table columns by specifying:

The position of each field within a row in the data file.

The column type, which is used to convert the field data type to the desired column data type.

If no column is mapped to a field, the field is not copied into the generated row(s). This behavior allows a data file to generate rows with different columns (in different tables).

Similarly, for bulk exporting data from a table, each <COLUMN> in the format file maps the column from the input table row to its corresponding field in the output data file.

When an XML document is validated through the XML Schema Definition (XSD) language, the xsi:type value is not put into the data set. However, you can put the xsi:type information into the data set by loading the XML format file into an XML document (for example, myDoc

), as illustrated in the following code snippet:

...; myDoc.LoadXml(xmlFormat);

XmlNodeList ColumnList = myDoc.GetElementsByTagName("COLUMN"); for(int i=0;i<ColumnList.Count;i++)

{

Console.Write("COLUMN: xsi:type=" +ColumnList[i].Attributes["type",

"http://www.w3.org/2001/XMLSchema-instance"].Value+"\n");

}

Sample XML Format Files

This section contains information on using XML format files in a variety of cases, including an Adventure Works example.

In the data files shown in the following examples,

<tab>

indicates a tab character in a data file, and

indicates a carriage return.

The examples illustrate key aspects of using XML format files, as follows:

Ordering character-data fields the same as table columns

Ordering data fields and table columns differently

Omitting a data field

Mapping different types of fields to columns

Mapping XML data to a table

Importing fixed-length or fixed-width fields

Additional Example

For information about how to create format files, see Create a Format File (SQL Server)

The following example shows an XML format file that describes a data file containing three fields of character data. The format file maps the data file to a table that contains three columns. The data fields correspond oneto-one with the columns of the table.

Table (row): Person (Age int, FirstName varchar(20), LastName varchar(30))

Data file (record): Age<tab>Firstname<tab>Lastname<return>

The following XML format file reads from the data file to the table.

In the

element, the format file represents the data values in all three fields as character data. For each field, the

TERMINATOR

attribute indicates the terminator that follows the data value.

The data fields correspond one-to-one with the columns of the table. In the

<ROW>

element, the format file maps the column

Age

to the first field, the column

FirstName

to the second field, and the column

LastName

to the third field.

<?xml version="1.0"?>

<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="\t"

MAX_LENGTH="12"/>

<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\t"

MAX_LENGTH="20" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\r\n"

MAX_LENGTH="30"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

</ROW>

</BCPFORMAT>

For an equivalent AdventureWorks2012 example, see Create a Format File (SQL Server)

The following example shows an XML format file that describes a data file containing three fields of character data. The format file maps the data file to a table that contains three columns that are ordered differently from the fields of the data file.

Table (row): Person (Age int, FirstName varchar(20), LastName varchar(30))

Data file (record): Age<tab>Lastname<tab>Firstname<return>

In the

element, the format file represents the data values in all three fields as character data.

In the

<ROW>

element, the format file maps the column

Age

to the first field, the column

FirstName

to the third field, and the column

LastName

to the second field.

<?xml version="1.0"?>

<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="\t"

MAX_LENGTH="12"/>

<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="20"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\r\n"

MAX_LENGTH="30" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

</ROW>

</BCPFORMAT>

For an equivalent AdventureWorks2012 example, see Use a Format File to Map Table Columns to Data-File Fields (SQL

For an equivalent AdventureWorks2012 example, see Use a Format File to Skip a Data Field (SQL Server)

The following example shows an XML format file that describes a data file containing four fields of character data. The format file maps the data file to a table that contains three columns. The second data field does not correspond to any table column.

Table (row): Person (Age int, FirstName Varchar(20), LastName Varchar(30))

Data file (record): Age<tab>employeeID<tab>Firstname<tab>Lastname<return>

In the

element, the format file represents the data values in all four fields as character data. For each field, the TERMINATOR attribute indicates the terminator that follows the data value.

In the

<ROW>

element, the format file maps the column

Age

to the first field, the column

FirstName

to the third field, and the column

LastName

to the fourth field.

<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="\t"

MAX_LENGTH="12"/>

<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\t"

MAX_LENGTH="10"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\t"

MAX_LENGTH="20"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

<FIELD ID="4" xsi:type="CharTerm" TERMINATOR="\r\n"

MAX_LENGTH="30"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

</ROW>

</BCPFORMAT>

The following example shows different types of fields and their mappings to columns.

<?xml version = "1.0"?>

<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<FIELD xsi:type="CharTerm" ID="C1" TERMINATOR="\t"

MAX_LENGTH="4"/>

<FIELD xsi:type="CharFixed" ID="C2" LENGTH="10"

COLLATION="SQL_LATIN1_GENERAL_CP1_CI_AS"/>

<FIELD xsi:type="CharPrefix" ID="C3" PREFIX_LENGTH="2"

MAX_LENGTH="32" COLLATION="SQL_LATIN1_GENERAL_CP1_CI_AS"/>

<FIELD xsi:type="NCharTerm" ID="C4" TERMINATOR="\t"

MAX_LENGTH="4"/>

<FIELD xsi:type="NCharFixed" ID="C5" LENGTH="10"

COLLATION="SQL_LATIN1_GENERAL_CP1_CI_AS"/>

<FIELD xsi:type="NCharPrefix" ID="C6" PREFIX_LENGTH="2"

MAX_LENGTH="32" COLLATION="SQL_LATIN1_GENERAL_CP1_CI_AS"/>

</RECORD>

<ROW>

<COLUMN SOURCE="C2" NAME="FirstName" xsi:type="SQLVARYCHAR"

LENGTH="16" NULLABLE="NO"/>

<COLUMN SOURCE="C7" NAME="Interest"xsi:type="SQLDECIMAL"

PRECISION="5" SCALE="3"/>

</ROW>

</BCPFORMAT>

The following example creates an empty two-column table ( t_xml

), in which the first column maps to the int data type and the second column maps to the xml

data type.

CREATE TABLE t_xml (c1 int, c2 xml)

The following XML format file would load a data file into table t_xml

<?xml version="1.0"?>

</RECORD>

<ROW>

</ROW>

</BCPFORMAT>

The following example describes fixed fields of

characters each. The format file represents these field lengths/widths as

LENGTH="10"

and

LENGTH="6"

, respectively. Every row of the data files ends with a carriage return-line feed combination, {CR}{LF}, which the format file represents as

TERMINATOR="\r\n"

<?xml version="1.0"?>

<BCPFORMAT

xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\r\n"

</RECORD>

<ROW>

</ROW>

</BCPFORMAT>

For additional examples of both non-XML format files and XML format files, see the following topics:

Use a Format File to Skip a Table Column (SQL Server)

Use a Format File to Skip a Data Field (SQL Server)

Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

Related Tasks

Use a Format File to Skip a Table Column (SQL Server)

Use a Format File to Skip a Data Field (SQL Server)

Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

Create a Format File (SQL Server)

3/24/2017 • 9 min to read • Edit Online

When you bulk import into a SQL Server table or bulk export data from a table, you can use a format file to a flexible system for writing data files that requires little or no editing to comply with other data formats or to read data files from other software programs.

SQL Server support two types of format file: non-XML format and XML format. The non-XML format is the original format that is supported by earlier versions of SQL Server.

Generally, XML and non-XML format files are interchangeable. However, we recommend that you use the XML syntax for new format files because they provide several advantages over non-XML format files.

The version of the bcp utility (Bcp.exe) used to read a format file must be the same as, or later than the version used to create the format file. For example, SQL Server 2012bcp can read a version 10.0 format file, which is generated by SQL

Server 2008bcp, but SQL Server 2008bcp cannot read a version 11.0 format file, which is generated by SQL Server

2012bcp.

This topic describes how to use the bcp utility to create a format file for a particular table. The format file is based on the data-type option specified (-n, -c, -w,or -N) and the table or view delimiters.

Creating a Non-XML Format File

To use a bcp command to create a format file, specify the format argument and use nul instead of a data-file path. The format option also requires the -f option, such as:

bcp table_or_view format nul -fformat_file_name

To distinguish a non-XML format file, we recommend that you use .fmt as the file name extension, for example,

MyTable.fmt.

For information about the structure and fields of non-XML format files, see Non-XML Format Files (SQL

For more information, see Non-XML Format Files (SQL Server)

This section contains the following examples that show how to use bcp commands to create a non-XML format file:

A. Creating a non-XML format file for native data

B. Creating a non-XML format file for character data

C. Creating a non-XML format file for Unicode native data

D. Creating a non-XML format file for Unicode character data

F. Using a format file with the code page option

The examples use the

HumanResources.Department

table in the AdventureWorks2012 sample database.

The

HumanResources.Department

table contains four columns:

DepartmentID

Name

GroupName

, and

ModifiedDate

The following example creates an XML format file,

Department-n.xml

, for the AdventureWorks2012

HumanResources.Department

table. The format file uses native data types. The contents of the generated format file are presented after the command.

The bcp command contains the following qualifiers.

QUALIFIERS

formatnul-f format_file

DESCRIPTION

Specifies the non-XML format file.

-n

-T

Specifies native data types.

Specifies that the bcp utility connects to SQL Server with a trusted connection using integrated security. If -T is not specified, you must specify -U and -P to successfully log in.

At the Windows command prompt, enter the following bcp

command: bcp AdventureWorks2012.HumanResources.Department format nul -T -n -f Department-n.fmt

The generated format file,

Department-n.fmt

, contains the following information:

12.0

1 SQLSMALLINT 0 2 "" 1 DepartmentID ""

2 SQLNCHAR 2 100 "" 2 Name SQL_Latin1_General_CP1_CI_AS

3 SQLNCHAR 2 100 "" 3 GroupName SQL_Latin1_General_CP1_CI_AS

4 SQLDATETIME 0 8 "" 4 ModifiedDate ""

The following example creates an XML format file,

Department.fmt

, for the AdventureWorks2012

HumanResources.Department

table. The format file uses character data formats and a non-default field terminator (

). The contents of the generated format file are presented after the command.

The bcp command contains the following qualifiers.

QUALIFIERS

formatnul-f format_file

DESCRIPTION

Specifies a non-XML format file.

-c

-T

Specifies character data.

Specifies that the bcp utility connects to SQL Server with a trusted connection using integrated security. If -T is not specified, you must specify -U and -P to successfully log in.

At the Windows command prompt, enter the following bcp

command:

bcp AdventureWorks2012.HumanResources.Department format nul -c -f Department-c.fmt -T

The generated format file,

Department-c.fmt

, contains the following information:

12.0

1 SQLCHAR 0 7 "\t" 1 DepartmentID ""

2 SQLCHAR 0 100 "\t" 2 Name SQL_Latin1_General_CP1_CI_AS

3 SQLCHAR 0 100 "\t" 3 GroupName SQL_Latin1_General_CP1_CI_AS

4 SQLCHAR 0 24 "\r\n" 4 ModifiedDate ""

For more information, see Non-XML Format Files (SQL Server)

To create a non-XML format file for Unicode native data for the

HumanResources.Department

table, use the following command: bcp AdventureWorks2012.HumanResources.Department format nul -T -N -f Department-n.fmt

For more information about how to use Unicode native data, see Use Unicode Native Format to Import or

Export Data (SQL Server) .

To create a non-XML format file for Unicode character data for the

HumanResources.Department

table that uses default terminators, use the following command: bcp AdventureWorks2012.HumanResources.Department format nul -T -w -f Department-w.fmt

For more information about how to use Unicode character data, see Use Unicode Character Format to Import or Export Data (SQL Server) .

If you create a format file using the bcp command (i.e. using “ bcp forma t …” ) information about the collation/code page will be written in the format file.

The following example format file for a table with 5 columns includes the collation.

13.0

1 SQLCHAR 0 0 "**\t**" 1 c_0 Cyrillic_General_CS_AS

2 SQLCHAR 0 0 "**\t**" 2 c_1 Cyrillic_General_CS_AS

3 SQLCHAR 0 3000 "**\t**" 3 c_2 Cyrillic_General_CS_AS

4 SQLCHAR 0 5 "**\t**" 4 c_3 ""

5 SQLCHAR 0 41 "!!!\r\r\n" 5 c_4 ""

If you try to import data into SQL Server using bcp in –c –C65001 –f format_file

…” or “

BULK INSERT

OPENROWSET

…

FORMATFILE='format_file' CODEPAGE=65001

…”, information about the collation/code page will have priority over 65001 option.

Therefore, if you generate a format file, you must manually delete the collation info from the generated format file before you start importing data back into SQL Server.

The following is an example of the format file without the collation info.

13.0

1 SQLCHAR 0 0 "**\t**" 1 c_0 ""

2 SQLCHAR 0 0 "**\t**" 2 c_1 ""

3 SQLCHAR 0 3000 "**\t**" 3 c_2 ""

4 SQLCHAR 0 5 "**\t**" 4 c_3 ""

5 SQLCHAR 0 41 "!!!\r\r\n" 5 c_4 ""

Creating an XML Format File

To use a bcp command to create a format file, specify the format argument and use nul instead of a data-file path. The format option always requires the -f option, and to create an XML format file, you must also specify the -x option, such as:

bcp table_or_view format nul-f format_file_name -x

To distinguish an XML format file, we recommend that you use .xml as the file name extension, for example,

MyTable.xml.

For information about the structure and fields of XML format files, see XML Format Files (SQL Server)

This section contains the following examples that show how to use bcp commands to create an XML format file:

A. Creating an XML format file for character data

B. Creating an XML format file for native data

The examples use the

HumanResources.Department

table in the AdventureWorks2012 sample database.

The

HumanResources.Department

table contains four columns:

DepartmentID

Name

GroupName

, and

ModifiedDate

Adventure Works Cycles is a fictional manufacturing company used to demonstrate database concepts and scenarios.

The following example creates an XML format file,

Department.xml

, for the AdventureWorks2012

HumanResources.Department

table. The format file uses character data formats and a non-default field terminator (

). The contents of the generated format file are presented after the command.

The bcp command contains the following qualifiers.

QUALIFIERS

formatnul-f format_file -x

-c

DESCRIPTION

Specifies the XML format file.

Specifies character data.

QUALIFIERS

-t

-T

DESCRIPTION

Specifies a comma (,) as the field terminator.

Note: If the data file uses the default field terminator (

), the -t switch is unnecessary.

Specifies that the bcp utility connects to SQL Server with a trusted connection using integrated security. If -T is not specified, you must specify -U and -P to successfully log in.

At the Windows command prompt, enter the following bcp

command: bcp AdventureWorks2012.HumanResources.Department format nul -c -x -f Department-c..xml –t, -T

The generated format file,

Department-c.xml

, contains the following XML elements:

<?xml version="1.0"?>

<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="100"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="100"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

</ROW>

</BCPFORMAT>

For information about the syntax of this format file, see XML Format Files (SQL Server)

. For information about

character data, see Use Character Format to Import or Export Data (SQL Server)

The following example creates an XML format file,

Department-n.xml

, for the

HumanResources.Department

table.

The format file uses native data types. The contents of the generated format file are presented after the command.

The bcp command contains the following qualifiers.

-n

-T

QUALIFIERS

formatnul-f format_file -x

DESCRIPTION

Specifies the XML format file.

Specifies native data types.

Specifies that the bcp utility connects to SQL Server with a trusted connection using integrated security. If -T is not specified, you must specify -U and -P to successfully log in.

At the Windows command prompt, enter the following bcp

command:

bcp AdventureWorks2012.HumanResources.Department format nul -x -f Department-n..xml -n -T

The generated format file,

Department-n.xml

, contains the following XML elements:

<?xml version="1.0"?>

<FIELD ID="2" xsi:type="NCharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

<FIELD ID="3" xsi:type="NCharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

</ROW>

</BCPFORMAT>

For information about the syntax of this format file, see XML Format Files (SQL Server)

. For information about

how to use native data, see Use Native Format to Import or Export Data (SQL Server)

Mapping Data Fields to Table Columns

As created by bcp, a format file describes all the table columns in order. You can modify a format file to rearrange or omit table rows. This lets you customize a format file to a data file whose fields do not map directly to the table columns. For more information, see the following topics:

Use a Format File to Skip a Table Column (SQL Server)

Use a Format File to Skip a Data Field (SQL Server)

Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

Use a Format File to Bulk Import Data (SQL Server)

3/24/2017 • 6 min to read • Edit Online

THIS TOPIC APPLIES TO: SQL Server (starting with 2008) Azure SQL Database Azure SQL Data

Warehouse Parallel Data Warehouse

This topic illustrates the use of a format file in bulk-import operations. A format file maps the fields of the data

file to the columns of the table. Please review Create a Format File (SQL Server)

for additional information.

OUTLINE

Before You Begin

Creating the Format Files

● Creating a Non-XML Format File

● Creating an XML Format File

Using a Format File to Bulk Import Data

● Using bcp and Non-XML Format File

● Using bcp and XML Format File

● Using BULK INSERT and Non-XML Format File

● Using BULK INSERT and XML Format File

● Using OPENROWSET(BULK...) and Non-XML Format File

● Using OPENROWSET(BULK...) and XML Format File

Before You Begin

For a format file to work with a Unicode character data file, all input fields must be Unicode text strings (that is, either fixed-size or character-terminated Unicode strings).

To bulk export or import SQLXML

data, use one of the following data types in your format file:

SQLCHAR or SQLVARYCHAR (the data is sent in the client code page or in the code page implied by the collation)

SQLNCHAR or SQLNVARCHAR (the data is sent as Unicode)

SQLBINARY or SQLVARYBIN (the data is sent without any conversion).

Azure SQL Database and Azure SQL Data Warehouse only support bcp . For additional information, see:

Load data into Azure SQL Data Warehouse

Load data from SQL Server into Azure SQL Data Warehouse (flat files)

Migrate Your Data

Example Test Conditions

The examples of format files in this topic are based on the table and data file defined below.

Sample Table

The script below creates a test database and a table named myFirstImport

. Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

CREATE DATABASE TestDatabase;

USE TestDatabase;

CREATE TABLE dbo.MyFirstImport (

PersonID smallint,

FirstName varchar(25),

LastName varchar(30),

BirthDate Date

);

Sample Data File

Using Notepad, create an empty file

D:\BCP\myFirstImport.bcp

and insert the following data:

1,Anthony,Grosse,1980-02-23

2,Alica,Fatnowna,1963-11-14

3,Stella,Rosenhain,1992-03-02

Alternatively, you can execute the following PowerShell script to create and populate the data file: cls

# revise directory as desired

$dir = 'D:\BCP\';

$bcpFile = $dir + 'MyFirstImport.bcp';

# Confirm directory exists

IF ((Test-Path -Path $dir) -eq 0)

{

Write-Host "The path $dir does not exist; please create or modify the directory.";

RETURN;

};

# clear content, will error if file does not exist, can be ignored

Clear-Content -Path $bcpFile -ErrorAction SilentlyContinue;

# Add data

Add-Content -Path $bcpFile -Value '1,Anthony,Grosse,1980-02-23';

Add-Content -Path $bcpFile -Value '2,Alica,Fatnowna,1963-11-14';

Add-Content -Path $bcpFile -Value '3,Stella,Rosenhain,1992-03-02';

#Review content

Get-Content -Path $bcpFile;

Invoke-Item $bcpFile;

Creating the Format Files

SQL Server support two types of format file: non-XML format and XML format. The non-XML format is the original format that is supported by earlier versions of SQL Server.

Please review Non-XML Format Files (SQL Server)

for detailed information. The following command will use the bcp utility to generate a non-xml format file, myFirstImport.fmt

, based on the schema of myFirstImport

data, t, is used to specify a comma as a field terminator

, and T is used to specify a trusted connection using integrated security. At a command prompt, enter the following command:

bcp TestDatabase.dbo.myFirstImport format nul -c -f D:\BCP\myFirstImport.fmt -t, -T

REM Review file

Notepad D:\BCP\myFirstImport.fmt

Your non-XML format file,

D:\BCP\myFirstImport.fmt

should look as follows:

13.0

1 SQLCHAR 0 7 "," 1 PersonID ""

2 SQLCHAR 0 25 "," 2 FirstName

SQL_Latin1_General_CP1_CI_AS

3 SQLCHAR 0 30 "," 3 LastName

SQL_Latin1_General_CP1_CI_AS

4 SQLCHAR 0 11 "\r\n" 4 BirthDate ""

Ensure your non-XML format file ends with a carriage return\line feed. Otherwise you will likely receive the following error message:

SQLState = S1000, NativeError = 0

Error = [Microsoft][ODBC Driver 13 for SQL Server]I/O error while reading BCP format file

Creating an XM L Format File

Please review XML Format Files (SQL Server)

for detailed information. The following command will use the bcp utility to create an xml format file, myFirstImport.xml

, based on the schema of myFirstImport

. To use a bcp command to create a format file, specify the format argument and use nul instead of a data-file path. The format option always requires the -f option, and to create an XML format file, you must also specify the -x option. In addition, for this example, the qualifier c is used to specify character data, t, is used to specify a

comma as a field terminator

, and T is used to specify a trusted connection using integrated security. At a command prompt, enter the following command: bcp TestDatabase.dbo.myFirstImport format nul -c -x -f D:\BCP\myFirstImport.xml -t, -T

REM Review file

Notepad D:\BCP\myFirstImport.xml

Your XML format file,

D:\BCP\myFirstImport.xml

should look as follows:

\<?xml version="1.0"?>

\<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

\<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="7"/>

\<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="25"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

\<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="30"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

\<FIELD ID="4" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="11"/>

</RECORD>

<ROW>

\<COLUMN SOURCE="1" NAME="PersonID" xsi:type="SQLSMALLINT"/>

\<COLUMN SOURCE="2" NAME="FirstName" xsi:type="SQLVARYCHAR"/>

\<COLUMN SOURCE="3" NAME="LastName" xsi:type="SQLVARYCHAR"/>

\<COLUMN SOURCE="4" NAME="BirthDate" xsi:type="SQLDATE"/>

</ROW>

</BCPFORMAT>

Using a Format File to Bulk Import Data

The examples below use the database, datafile, and format files created above.

Using bcp

At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.MyFirstImport;"

REM Import data bcp TestDatabase.dbo.myFirstImport IN D:\BCP\myFirstImport.bcp -f D:\BCP\myFirstImport.fmt -T

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.MyFirstImport"

Using bcp

At a command prompt, enter the following command:

REM Truncate table (for testing)

SQLCMD -Q "TRUNCATE TABLE TestDatabase.dbo.MyFirstImport;"

REM Import data bcp TestDatabase.dbo.myFirstImport IN D:\BCP\myFirstImport.bcp -f D:\BCP\myFirstImport.xml -T

REM Review results

SQLCMD -Q "SELECT * FROM TestDatabase.dbo.MyFirstImport;"

Using B ULK INSERT

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myFirstImport; -- (for testing)

BULK INSERT dbo.myFirstImport

FROM 'D:\BCP\myFirstImport.bcp'

WITH (FORMATFILE = 'D:\BCP\myFirstImport.fmt');

-- review results

SELECT * FROM TestDatabase.dbo.myFirstImport;

Using B ULK INSERT

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myFirstImport; -- (for testing)

BULK INSERT dbo.myFirstImport

FROM 'D:\BCP\myFirstImport.bcp'

WITH (FORMATFILE = 'D:\BCP\myFirstImport.xml');

-- review results

SELECT * FROM TestDatabase.dbo.myFirstImport;

Using OPENROWSET(B ULK...)

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myFirstImport; -- (for testing)

INSERT INTO dbo.myFirstImport

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myFirstImport.bcp',

FORMATFILE = 'D:\BCP\myFirstImport.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myFirstImport;

Using OPENROWSET(B ULK...)

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myFirstImport; -- (for testing)

INSERT INTO dbo.myFirstImport

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myFirstImport.bcp',

FORMATFILE = 'D:\BCP\myFirstImport.xml'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myFirstImport;

More examples!

Use a Format File to Skip a Table Column (SQL Server)

Use a Format File to Skip a Data Field (SQL Server)

Use a Format File to Map Table Columns to Data-File Fields (SQL Server)

Use a Format File to Skip a Table Column (SQL

Server)

3/24/2017 • 6 min to read • Edit Online

This topic describes format files. You can use a format file to skip importing a table column when the field does not exist in the data file. A data file can contain fewer fields than the number of columns in the table only if the skipped columns are nullable and/or have default value.

Sample Table and Data File

The following examples require a table named myTestSkipCol

in the AdventureWorks2012 sample database under the dbo schema. Create this table as follows:

USE AdventureWorks2012;

CREATE TABLE myTestSkipCol

(

Col1 smallint,

Col2 nvarchar(50) NULL,

Col3 nvarchar(50) not NULL

);

The following examples use a sample data file, myTestSkipCol2.dat

, which contains only two fields, although the corresponding table contains three columns:

1,DataForColumn3

To bulk import data from myTestSkipCol2.dat

into the myTestSkipCol

table, the format file must map the first data field to

Col1

, the second field to

Col3

, skipping

Col2

Using a Non-XML Format File

You can modify a non-XML format file to skip a table column. Typically, this involves using the bcp utility to create a default non-XML format file and the modifying the default file in a text editor. The modified format file must map each existing fields to its corresponding table column and indicate which table column or columns to skip. Two alternatives exist for modifying a default non-XML data file. Either alternative indicates that the data field does not exist in the data file and that no data will be inserted into the corresponding table column.

This topic uses the default non-XML format file that was created for the myTestSkipCol

sample table by using the following bcp command: bcp AdventureWorks2012..myTestSkipCol format nul -f myTestSkipCol_Default.fmt -c -T

The previous command creates a non-XML format file, myTestSkipCol_Default.fmt

. This format file is called a

default format file because it is the form generated by bcp. Typically, a default format file describes a one-to-

one correspondence between data-file fields and table columns.

You might have to specify the name of the server instance to which you are connecting. Also, you might have to specify the user name and password. For more information, see bcp Utility .

The following illustration shows the values in this sample default format files. The illustration also shows the name of each format-file field.

For more information about the format-file fields, see

Non-XML Format Files (SQL Server)

To skip a table column, edit the default non-XML format file and modify the file by using one of the following alternative methods:

The preferred method involves three basic steps. First, delete any format-file row that describes a field that is missing from the data file. Then, reduce the "Host file field order" value of each format-file row that follows a deleted row. The goal is sequential "Host file field order" values, 1 through n, that reflect the actual position of each data field in the data file. Finally, reduce the value in the "Number of columns" field to reflect the actual number of fields in the data file.

The following example is based on the default format file for the myTestSkipCol

table, which is created in

"Creating a Default Non-XML Format File," earlier in this topic. This modified format file maps the first data field to

Col1

, skips

Col2

, and maps the second data field to

Col3

. The row for

Col2

has been deleted.

9.0

1 SQLCHAR 0 7 "\t" 1 Col1 ""

2 SQLCHAR 0 100 "\r\n" 3 Col3 SQL_Latin1_General_CP1_CI_AS

Alternatively, to skip a table column, you can modify the definition of the format-file row that corresponds to the table column. In this format-file row, the "prefix length," "host file data length," and

"server column order" values must be set to 0. Also, the "terminator" and "column collation" fields must be set to "" (NULL).

The "server column name" value requires a nonblank string, though the actual column name is not necessary. The remaining format fields require their default values.

The following example is also derived from the default format file for the myTestSkipCol

table.

9.0

1 SQLCHAR 0 7 "\t" 1 Col1 ""

2 SQLCHAR 0 0 "" 0 Col2 ""

3 SQLCHAR 0 100 "\r\n" 3 Col3 SQL_Latin1_General_CP1_CI_AS

The following examples are also based on the myTestSkipCol

sample table and the myTestSkipCol2.dat

sample data file that are created in "Sample Table and Data File," earlier in this topic.

This example works by using either of the modified non-XML format files created in "Methods for Modifying a

Non-XML Format File," earlier in this topic. In this example, the modified format file is named

C:\myTestSkipCol2.fmt

. To use

BULK INSERT

to bulk import the myTestSkipCol2.dat

data file, in the SQL Server

Management Studio Query Editor, execute the following code:

USE AdventureWorks2012;

BULK INSERT myTestSkipCol

FROM 'C:\myTestSkipCol2.dat'

WITH (FORMATFILE = 'C:\myTestSkipCol2.fmt');

SELECT * FROM myTestSkipCol;

Using an XML Format File

With an XML format file, you cannot skip a column when you are importing directly into a table by using a bcp command or a BULK INSERT statement. However, you can import into all but the last column of a table. If you have to skip any but the last column, you must create a view of the target table that contains only the columns contained in the data file. Then, you can bulk import data from that file into the view.

To use an XML format file to skip a table column by using OPENROWSET(BULK...), you have to provide explicit list of columns in the select list and also in the target table, as follows:

INSERT ... SELECT FROM OPENROWSET(BULK...)

The examples of modified format files are based on the myTestSkipCol

sample table and data file that are created in "Sample Table and Data File," earlier in this topic. The following bcp command creates a default XML format file for the myTestSkipCol

table: bcp AdventureWorks2012..myTestSkipCol format nul -f myTestSkipCol_Default.xml -c -x -T

The resulting default non-XML format file describes a one-to-one correspondence between data-file fields and table columns, as follows:

\<?xml version="1.0"?>

\<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

\<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="7"/>

\<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="100"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

\<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="100"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

\<COLUMN SOURCE="1" NAME="Col1" xsi:type="SQLSMALLINT"/>

\<COLUMN SOURCE="2" NAME="Col2" xsi:type="SQLNVARCHAR"/>

\<COLUMN SOURCE="3" NAME="Col3" xsi:type="SQLNVARCHAR"/>

</ROW>

</BCPFORMAT>

For information about the structure of XML format files, see

XML Format Files (SQL Server)

The examples in this section use the myTestSkipCol

sample table and the myTestSkipCol2.dat

sample data file that are created in "Sample Table and Data File," earlier in this topic. To import the data from myTestSkipCol2.dat

into the myTestSkipCol

table, the examples use the following modified XML format file, myTestSkipCol2-x.xml

. This is based on the format file that is created in "Creating a Default XML Format File," earlier in this topic.

\<?xml version="1.0"?>

\<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

\<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="7"/>

\<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="100"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

\<COLUMN SOURCE="1" NAME="Col1" xsi:type="SQLSMALLINT"/>

\<COLUMN SOURCE="2" NAME="Col3" xsi:type="SQLNVARCHAR"/>

</ROW>

</BCPFORMAT>

The following example uses the

OPENROWSET

bulk rowset provider and the myTestSkipCol2.xml

format file. The example bulk imports the myTestSkipCol2.dat

data file into the myTestSkipCol

table. The statement contains an explicit list of columns in the select list and also in the target table, as required.

In the SQL Server Management Studio Query Editor, execute the following code:

USE AdventureWorks2012;

INSERT INTO myTestSkipCol

(Col1,Col3)

SELECT Col1,Col3

FROM OPENROWSET(BULK 'C:\myTestSkipCol2.Dat',

FORMATFILE='C:\myTestSkipCol2.Xml'

) as t1 ;

The following example creates the v_myTestSkipCol

on the myTestSkipCol

table. This view skips the second table column,

Col2

. The example then uses

BULK INSERT

to import the myTestSkipCol2.dat

data file into this view.

In the SQL Server Management Studio Query Editor, execute the following code:

CREATE VIEW v_myTestSkipCol AS

SELECT Col1,Col3

FROM myTestSkipCol;

USE AdventureWorks2012;

BULK INSERT v_myTestSkipCol

FROM 'C:\myTestSkipCol2.dat'

WITH (FORMATFILE='C:\myTestSkipCol2.xml');

Use a Format File to Skip a Data Field (SQL Server)

3/24/2017 • 6 min to read • Edit Online

A data file can contain more fields than the number of columns in the table. This topic describes modifying both non-XML and XML format files to accommodate a data file with more fields by mapping the table columns to

the corresponding data fields and ignoring the extra fields. Please review Create a Format File (SQL Server)

for additional information.

OUTLINE

Creating the Format Files

● Creating a Non-XML Format File

● Modifying a Non-XML Format File

● Creating an XML Format File

● Modifying an XML Format File

Importing Data with a Format File to skip a Data Field

● Using bcp and Non-XML Format File

● Using bcp and XML Format File

● Using BULK INSERT and Non-XML Format File

● Using BULK INSERT and XML Format File

● Using OPENROWSET(BULK...) and Non-XML Format File

● Using OPENROWSET(BULK...) and XML Format File

Either a non-XML or XML format file can be used to bulk import a data file into the table by using a bcp utility command,

BULK INSERT statement, or INSERT ... SELECT * FROM OPENROWSET(BULK...) statement. For more information, see Use a

Format File to Bulk Import Data (SQL Server) .

Example Test Conditions

The examples of modified format files in this topic are based on the table and data file defined below.

The script below creates a test database and a table named myTestSkipField

. Execute the following Transact-

SQL in Microsoft SQL Server Management Studio (SSMS):

CREATE DATABASE TestDatabase;

USE TestDatabase;

CREATE TABLE myTestSkipField

(

PersonID smallint,

FirstName varchar(25),

LastName varchar(30)

);

Create an empty file

D:\BCP\myTestSkipField.bcp

and insert the following data:

1,SkipMe,Anthony,Grosse

2,SkipMe,Alica,Fatnowna

3,SkipMe,Stella,Rosenhain

Creating the Format Files

To bulk import data from myTestSkipField.bcp

into the myTestSkipField

table, the format file must do the following:

Map the first data field to the first column,

PersonID

Skip the second data field.

Map the third data field to the second column,

FirstName

Map the fourth data field to the third column,

LastName

The simplest method to create the format file is by using the bcp utility . First, create a base format file from the existing table. Second, modify the base format file to reflect the actual data file.

Please review Non-XML Format Files (SQL Server)

for detailed information. The following command will use the bcp utility to generate a non-xml format file, myTestSkipField.fmt

, based on the schema of myTestSkipField

. In addition, the qualifier c

is used to specify character data , t,

is used to specify a comma as a field terminator, and

is used to specify a trusted connection using integrated security. At a command prompt, enter the following command: bcp TestDatabase.dbo.myTestSkipField format nul -c -f D:\BCP\myTestSkipField.fmt -t, -T

See Structure of Non-XML Format Files

for terminology. Open

D:\BCP\myTestSkipField.fmt

in Notepad and perform the following modifications: 1) Copy the entire format-file row for

FirstName

and paste it directly after

FirstName

on the next line. 2) Increase the host file field order value by one for the new row and all subsequent rows. 3) Increase the number of columns value to reflect the actual number of fields in the data file. 3) Modify the server column order from

for the second format-file row.

Compare the changes made:

Before

13.0

1 SQLCHAR 0 7 "," 1 PersonID ""

2 SQLCHAR 0 25 "," 2 FirstName SQL_Latin1_General_CP1_CI_AS

3 SQLCHAR 0 30 "\r\n" 3 LastName SQL_Latin1_General_CP1_CI_AS

After

13.0

1 SQLCHAR 0 7 "," 1 PersonID ""

2 SQLCHAR 0 25 "," 0 FirstName SQL_Latin1_General_CP1_CI_AS

3 SQLCHAR 0 25 "," 2 FirstName SQL_Latin1_General_CP1_CI_AS

4 SQLCHAR 0 50 "\r\n" 3 LastName SQL_Latin1_General_CP1_CI_AS

The modified format file now reflects:

4 data fields

The first data field in myTestSkipField.bcp

is mapped to the first column, myTestSkipField.. PersonID

The second data field in myTestSkipField.bcp

is not mapped to any column.

The third data field in myTestSkipField.bcp

is mapped to the second column, myTestSkipField.. FirstName

The fourth data field in myTestSkipField.bcp

is mapped to the third column, myTestSkipField.. LastName

Please review XML Format Files (SQL Server)

for detailed information. The following command will use the bcp utility to create an xml format file, myTestSkipField.xml

, based on the schema of myTestSkipField

. In addition, the qualifier c

is used to specify character data , t,

is used to specify a comma as a field terminator, and

T is used to specify a trusted connection using integrated security. The x

qualifier must be used to generate an

XML-based format file. At a command prompt, enter the following command: bcp TestDatabase.dbo.myTestSkipField format nul -c -x -f D:\BCP\myTestSkipField.xml -t, -T

See Schema Syntax for XML Format Files

for terminology. Open

D:\BCP\myTestSkipField.xml

in Notepad and perform the following modifications: 1) Copy the entire second field and paste it directly after the second field on the next line. 2) Increase the "FIELD ID" value by 1 for the new FIELD and for each subsequent FIELD. 3)

Increase the "COLUMN SOURCE" value by 1 for

FirstName

, and

LastName

to reflect the revised mapping.

Compare the changes made:

Before

\<?xml version="1.0"?>

\<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

\<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="7"/>

\<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="25"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

\<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="30"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

\<COLUMN SOURCE="1" NAME="PersonID" xsi:type="SQLSMALLINT"/>

\<COLUMN SOURCE="2" NAME="FirstName" xsi:type="SQLVARYCHAR"/>

\<COLUMN SOURCE="3" NAME="LastName" xsi:type="SQLVARYCHAR"/>

</ROW>

</BCPFORMAT>

After

\<?xml version="1.0"?>

\<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

\<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="7"/>

\<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="25"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

\<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="25"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

\<FIELD ID="4" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="30"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

\<COLUMN SOURCE="1" NAME="PersonID" xsi:type="SQLSMALLINT"/>

\<COLUMN SOURCE="3" NAME="FirstName" xsi:type="SQLVARYCHAR"/>

\<COLUMN SOURCE="4" NAME="LastName" xsi:type="SQLVARYCHAR"/>

</ROW>

</BCPFORMAT>

The modified format file now reflects:

4 data fields

FIELD 1 which corresponds to COLUMN 1 is mapped to the first table column, myTestSkipField.. PersonID

FIELD 2 does not correspond to any COLUMN and thus, is not mapped to any table column.

FIELD 3 which corresponds to COLUMN 3 is mapped to the second table column, myTestSkipField.. FirstName

FIELD 4 which corresponds to COLUMN 4 is mapped to the third table column, myTestSkipField.. LastName

Importing Data with a Format File to skip a Data Field

The examples below use the database, datafile, and format files created above.

Using bcp

At a command prompt, enter the following command: bcp TestDatabase.dbo.myTestSkipField IN D:\BCP\myTestSkipField.bcp -f D:\BCP\myTestSkipField.fmt -T

Using bcp

At a command prompt, enter the following command: bcp TestDatabase.dbo.myTestSkipField IN D:\BCP\myTestSkipField.bcp -f D:\BCP\myTestSkipField.xml -T

Using B ULK INSERT

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myTestSkipField;

BULK INSERT dbo.myTestSkipField

FROM 'D:\BCP\myTestSkipField.bcp'

WITH (FORMATFILE = 'D:\BCP\myTestSkipField.fmt');

-- review results

SELECT * FROM TestDatabase.dbo.myTestSkipField;

Using B ULK INSERT

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myTestSkipField;

BULK INSERT dbo.myTestSkipField

FROM 'D:\BCP\myTestSkipField.bcp'

WITH (FORMATFILE = 'D:\BCP\myTestSkipField.xml');

-- review results

SELECT * FROM TestDatabase.dbo.myTestSkipField;

Using OPENROWSET(B ULK...)

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myTestSkipField;

INSERT INTO dbo.myTestSkipField

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myTestSkipField.bcp',

FORMATFILE = 'D:\BCP\myTestSkipField.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myTestSkipField;

Using OPENROWSET(B ULK...)

their corresponding table columns. Please review Create a Format File (SQL Server)

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myTestSkipField;

INSERT INTO dbo.myTestSkipField

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myTestSkipField.bcp',

FORMATFILE = 'D:\BCP\myTestSkipField.xml'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myTestSkipField;

Use a Format File to Map Table Columns to Data-

File Fields (SQL Server)

3/24/2017 • 7 min to read • Edit Online

A data file can contain fields arranged in a different order from the corresponding columns in the table. This topic presents both non-XML and XML format files that have been modified to accommodate a data file whose fields are arranged in a different order from the table columns. The modified format file maps the data fields to

for additional information.

OUTLINE

Creating the Format Files

●

Sample Table

●

Sample Data File

●

Creating a Non-XML Format File

●

Modifying the Non-XML Format File

●

Creating an XML Format File

●

Modifying the XML Format File

Importing Data with a Format File to Map Table Columns to Data-File Field

●

Using bcp and Non-XML Format File

●

Using bcp and XML Format File

●

Using BULK INSERT and Non-XML Format File

●

Using BULK INSERT and XML Format File

●

Using OPENROWSET(BULK...) and Non-XML Format File

●

Using OPENROWSET(BULK...) and XML Format File

Either a non-XML or XML format file can be used to bulk import a data file into the table by using a bcp utility command, BULK INSERT statement, or INSERT ... SELECT * FROM OPENROWSET(BULK...) statement. For more

information, see Use a Format File to Bulk Import Data (SQL Server)

Example Test Conditions

The examples of modified format files in this topic are based on the table and data file defined below.

Sample Table

The script below creates a test database and a table named myRemap

. Execute the following Transact-SQL in

Microsoft SQL Server Management Studio (SSMS):

CREATE DATABASE TestDatabase;

USE TestDatabase;

CREATE TABLE myRemap

(

PersonID smallint,

FirstName varchar(25),

LastName varchar(30),

Gender char(1)

);

Sample Data File

The data below presents

FirstName

and

LastName

in the reverse order as presented in the table myRemap

Using Notepad, create an empty file

D:\BCP\myRemap.bcp

and insert the following data:

1,Grosse,Anthony,M

2,Fatnowna,Alica,F

3,Rosenhain,Stella,F

Creating the Format Files

To bulk import data from myRemap.bcp

into the myRemap

table, the format file must do the following:

Map the first data field to the first column,

PersonID

Map the second data field to the third column,

LastName

Map the third data field to the second column,

FirstName

Map the fourth data field to the fourth column,

Gender

Please review Non-XML Format Files (SQL Server)

for detailed information. The following command will use the bcp utility to generate a non-xml format file, myRemap.fmt

, based on the schema of myRemap

. In addition, the qualifier c

is used to specify character data, t,

is used to specify a comma as a field terminator, and

T is used to specify a trusted connection using integrated security. At a command prompt, enter the following command: bcp TestDatabase.dbo.myRemap format nul -c -f D:\BCP\myRemap.fmt -t, -T

See Structure of Non-XML Format Files

for terminology. Open

D:\BCP\myRemap.fmt

in Notepad and perform the following modifications: 1) Re-arrange the order of the format-file rows so that the rows are in the same order as the data in myRemap.bcp

. 2) Ensure the host file field order values are sequential. 3) Ensure there is a carriage return after the last format-file row.

Compare the changes:

Before

13.0

1 SQLCHAR 0 7 "," 1 PersonID ""

2 SQLCHAR 0 25 "," 2 FirstName SQL_Latin1_General_CP1_CI_AS

3 SQLCHAR 0 30 "," 3 LastName SQL_Latin1_General_CP1_CI_AS

4 SQLCHAR 0 1 "\r\n" 4 Gender SQL_Latin1_General_CP1_CI_AS

After

13.0

1 SQLCHAR 0 7 "," 1 PersonID ""

2 SQLCHAR 0 30 "," 3 LastName SQL_Latin1_General_CP1_CI_AS

3 SQLCHAR 0 25 "," 2 FirstName SQL_Latin1_General_CP1_CI_AS

4 SQLCHAR 0 1 "\r\n" 4 Gender SQL_Latin1_General_CP1_CI_AS

The modified format file now reflects:

The first data field in myRemap.bcp

is mapped to the first column, myRemap.. PersonID

The second data field in myRemap.bcp

is mapped to the third column, myRemap.. LastName

The third data field in myRemap.bcp

is mapped to the second column, myRemap.. FirstName

The fourth data field in myRemap.bcp

is mapped to the fourth column, myRemap.. Gender

Creating an XM L Format File

Please review XML Format Files (SQL Server)

for detailed information. The following command will use the bcp utility to create an xml format file, myRemap.xml

, based on the schema of myRemap

. In addition, the qualifier c is used to specify character data, t,

is used to specify a comma as a field terminator, and

is used to specify a trusted connection using integrated security. The x

qualifier must be used to generate an XML-based format file. At a command prompt, enter the following command: bcp TestDatabase.dbo.myRemap format nul -c -x -f D:\BCP\myRemap.xml -t, -T

See Schema Syntax for XML Format Files

for terminology. Open

D:\BCP\myRemap.xml

in Notepad and perform the following modifications: 1) The order in which the <FIELD> elements are declared in the format file is the order in which those fields appear in the data file, thus reverse the order for the <FIELD> elements with ID attributes 2 and 3. 2) Ensure the <FIELD> ID attribute values are sequential. 3) The order of the <COLUMN> elements in the <ROW> element defines the order in which they are returned by the bulk operation. The XML format file assigns each <COLUMN> element a local name that has no relationship to the column in the target table of a bulk import operation. The order of the <COLUMN> elements is independent of the order of

<FIELD> elements in a <RECORD> definition. Each <COLUMN> element corresponds to a <FIELD> element

(whose ID is specified in the SOURCE attribute of the <COLUMN> element). Thus, the values for <COLUMN>

SOURCE are the only attributes that require revision. Reverse the order for <COLUMN> SOURCE attributes 2 and 3.

Compare the changes

Before

\<?xml version="1.0"?>

\<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

\<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="7"/>

\<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="25"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

\<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="30"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

\<FIELD ID="4" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="1"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

\<COLUMN SOURCE="1" NAME="PersonID" xsi:type="SQLSMALLINT"/>

\<COLUMN SOURCE="2" NAME="FirstName" xsi:type="SQLVARYCHAR"/>

\<COLUMN SOURCE="3" NAME="LastName" xsi:type="SQLVARYCHAR"/>

\<COLUMN SOURCE="4" NAME="Gender" xsi:type="SQLCHAR"/>

</ROW>

</BCPFORMAT>

After

\<?xml version="1.0"?>

\<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

\<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="7"/>

\<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="30"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

\<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="25"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

\<FIELD ID="4" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="1"

COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

</RECORD>

<ROW>

\<COLUMN SOURCE="1" NAME="PersonID" xsi:type="SQLSMALLINT"/>

\<COLUMN SOURCE="3" NAME="FirstName" xsi:type="SQLVARYCHAR"/>

\<COLUMN SOURCE="2" NAME="LastName" xsi:type="SQLVARYCHAR"/>

\<COLUMN SOURCE="4" NAME="Gender" xsi:type="SQLCHAR"/>

</ROW>

</BCPFORMAT>

The modified format file now reflects:

FIELD 1, which corresponds to COLUMN 1, is mapped to the first table column, myRemap.. PersonID

FIELD 2, which corresponds to COLUMN 2, is re-mapped to the third table column, myRemap.. LastName

FIELD 3, which corresponds to COLUMN 3, is re-mapped to the second table column, myRemap.. FirstName

FIELD 4, which corresponds to COLUMN 4, is mapped to the fourth table column, myRemap.. Gender

Importing Data with a Format File to Map Table Columns to Data-

File Field

The examples below use the database, datafile, and format files created above.

Using bcp

At a command prompt, enter the following command: bcp TestDatabase.dbo.myRemap IN D:\BCP\myRemap.bcp -f D:\BCP\myRemap.fmt -T

Using bcp

At a command prompt, enter the following command: bcp TestDatabase.dbo.myRemap IN D:\BCP\myRemap.bcp -f D:\BCP\myRemap.xml -T

Using B ULK INSERT

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myRemap;

BULK INSERT dbo.myRemap

FROM 'D:\BCP\myRemap.bcp'

WITH (FORMATFILE = 'D:\BCP\myRemap.fmt');

-- review results

SELECT * FROM TestDatabase.dbo.myRemap;

Using B ULK INSERT

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myRemap;

BULK INSERT dbo.myRemap

FROM 'D:\BCP\myRemap.bcp'

WITH (FORMATFILE = 'D:\BCP\myRemap.xml');

-- review results

SELECT * FROM TestDatabase.dbo.myRemap;

Using OPENROWSET(B ULK...)

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myRemap;

INSERT INTO dbo.myRemap

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myRemap.bcp',

FORMATFILE = 'D:\BCP\myRemap.fmt'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myRemap;

Using OPENROWSET(B ULK...)

Import and Export Bulk Data by Using the bcp Utility (SQL Server).

Execute the following Transact-SQL in Microsoft SQL Server Management Studio (SSMS):

USE TestDatabase;

TRUNCATE TABLE myRemap;

INSERT INTO dbo.myRemap

SELECT *

FROM OPENROWSET (

BULK 'D:\BCP\myRemap.bcp',

FORMATFILE = 'D:\BCP\myRemap.xml'

) AS t1;

-- review results

SELECT * FROM TestDatabase.dbo.myRemap;

Examples of Bulk Import and Export of XML

Documents (SQL Server)

3/24/2017 • 7 min to read • Edit Online

THIS TOPIC APPLIES TO: SQL Server (starting with 2016) Azure SQL Database Azure SQL Data

Warehouse Parallel Data Warehouse

You can bulk import XML documents into a SQL Server database or bulk export them from a SQL Server database. This topic provides examples of both.

To bulk import data from a data file into a SQL Server table or non-partitioned view, you can use the following:

bcp utility

You can also use the bcp utility to export data from anywhere in a SQL Server database that a SELECT statement works, including partitioned views.

BULK INSERT

INSERT ... SELECT * FROM OPENROWSET(BULK...)

Note: For more information, see

Import Bulk Data by Using BULK INSERT or OPENROWSET (BULK...)(SQL Server).

How to import XML into SQL Server with the XML Bulk Load component.

XML Schema Collections (SQL Server)

Examples

The examples are the following:

A. Bulk importing XML data as a binary byte stream

B. Bulk importing XML data in an existing row

C. Bulk importing XML data from a file that contains a DTD

D. Specifying the field terminator explicitly using a format file

E. Bulk exporting XML data

Bulk importing XML data as a binary byte stream

When you bulk import XML data from a file that contains an encoding declaration that you want to apply, specify the SINGLE_BLOB option in the OPENROWSET(BULK…) clause. The SINGLE_BLOB option ensures that the XML parser in SQL Server imports the data according to the encoding scheme specified in the XML declaration.

To test example A below, create sample table

USE tempdb

CREATE TABLE T (IntCol int, XmlCol xml);

Before you can run example A, you must create a UTF-8 encoded file (

C:\SampleFolder\SampleData3.txt

) that contains the following sample instance that specifies the

UTF-8

encoding scheme.

\<?xml version="1.0" encoding="UTF-8"?>

<Root>

</ProductDescription>

</Root>

This example uses the

SINGLE_BLOB

option in an

INSERT ... SELECT * FROM OPENROWSET(BULK...)

statement to import data from a file named

SampleData3.txt

and insert an XML instance in the single-column table, sample table

INSERT INTO T(XmlCol)

SELECT * FROM OPENROWSET(

BULK 'c:\SampleFolder\SampleData3.txt',

SINGLE_BLOB) AS x;

By using SINGLE_BLOB in this case, you can avoid a mismatch between the encoding of the XML document (as specified by the XML encoding declaration) and the string codepage implied by the server.

If you use NCLOB or CLOB data types and run into a codepage or encoding conflict, you must do one of the following:

Remove the XML declaration to successfully import the contents of the XML data file.

Specify a code page in the CODEPAGE option of the query that matches the encoding scheme that is used in the XML declaration.

Match, or resolve, the database collation settings with a non-Unicode XML encoding scheme.

Bulk importing XML data in an existing row

This example uses the

OPENROWSET

bulk rowset provider to add an XML instance to an existing row or rows in sample table

To run this example, you must first complete the test script provided in example A. That example creates the tempdb.dbo.T

table and bulk imports data from

SampleData3.txt

Example B uses a modified version of the

SampleData3.txt

sample data file from the preceding example. To run this example, modify the content of this file as follows:

<Root>

</ProductDescription>

</Root>

-- Query before update shows initial state of XmlCol values.

SELECT * FROM T

UPDATE T

SET XmlCol =(

SELECT * FROM OPENROWSET(

BULK 'C:\SampleFolder\SampleData3.txt',

SINGLE_BLOB

) AS x

)

WHERE IntCol = 1;

Bulk importing XML data from a file that contains a DTD

We recommended that you not enable support for Document Type Definitions (DTDs) if it is not required in your XML environment. Turning on DTD support increases the attackable surface area of your server, and may expose it to a denialof-service attack. If you must enable DTD support, you can reduce this security risk by processing only trusted XML documents.

During an attempt to use a bcp command to import XML data from a file that contains a DTD, an error similar to the following can occur:

"SQLState = 42000, NativeError = 6359"

"Error = [Microsoft][SQL Server Native Client][ SQL Server]Parsing XML with internal subset DTDs not allowed.

Use CONVERT with style option 2 to enable limited internal subset DTD support."

"BCP copy %s failed"

To work around this problem, you can import XML data from a data file that contains a DTD by using the

OPENROWSET(BULK...)

function and then specifying the

CONVERT

option in the

SELECT

clause of the command. The basic syntax for the command is:

INSERT ... SELECT CONVERT(…) FROM OPENROWSET(BULK...)

Before you can test this bulk import example, create a file (

C:\temp\Dtdfile.xml

) that contains the following sample instance:

<!DOCTYPE DOC [<!ATTLIST elem1 attr1 CDATA "defVal1">]><elem1>January</elem1>

Example C uses the

sample table that is created by the following

CREATE TABLE

statement:

USE tempdb;

CREATE TABLE T1(XmlCol xml);

This example uses

OPENROWSET(BULK...)

and specifies the

CONVERT

option in the

SELECT

clause to import the XML data from

Dtdfile.xml

into sample table

INSERT T1

SELECT CONVERT(xml, BulkColumn, 2) FROM

OPENROWSET(Bulk 'c:\temp\Dtdfile.xml', SINGLE_BLOB) [rowsetresults];

After the

INSERT

statement executes, the DTD is stripped from the XML and stored in the

table.

When you bulk import or export an XML document, you should use a field terminator

Specifying the field terminator explicitly using a format file

The following example shows how to bulk import the following XML document,

Xmltable.dat

The document in

Xmltable.dat

contains two XML values, one for each row. The first XML value is encoded with

UTF-16, and the second value is encoded with UTF-8.

The contents of this data file are shown in the following Hex dump:

FF FE 3C 00 3F 00 78 00-6D 00 6C 00 20 00 76 00 *..\<.?.x.m.l. .v.*

65 00 72 00 73 00 69 00-6F 00 6E 00 3D 00 22 00 *e.r.s.i.o.n.=.".*

31 00 2E 00 30 00 22 00-20 00 65 00 6E 00 63 00 *1...0.". .e.n.c.*

6F 00 64 00 69 00 6E 00-67 00 3D 00 22 00 75 00 *o.d.i.n.g.=.".u.*

74 00 66 00 2D 00 31 00-36 00 22 00 3F 00 3E 00 *t.f.-.1.6.".?.>.*

3C 00 72 00 6F 00 6F 00-74 00 3E 00 A2 4F 9C 76 *\<.r.o.o.t.>..O.v*

0C FA 77 E4 80 00 89 00-00 06 90 06 91 2E 9B 2E *..w.............*

99 34 A2 34 86 00 83 02-92 20 7F 02 4E C5 E4 A3 *.4.4..... ..N...*

34 B2 B7 B3 B7 FE F8 FF-F8 00 3C 00 2F 00 72 00 *4.........\<./.r.*

6F 00 6F 00 74 00 3E 00-00 00 00 00 7A EF BB BF *o.o.t.>.....z...*

3C 3F 78 6D 6C 20 76 65-72 73 69 6F 6E 3D 22 31 *\<?xml version="1*

2E 30 22 20 65 6E 63 6F-64 69 6E 67 3D 22 75 74 *.0" encoding="ut*

66 2D 38 22 3F 3E 3C 72-6F 6F 74 3E E4 BE A2 E7 *f-8"?><root>....*

9A 9C EF A8 8C EE 91 B7-C2 80 C2 89 D8 80 DA 90 *................*

E2 BA 91 E2 BA 9B E3 92-99 E3 92 A2 C2 86 CA 83 *................*

E2 82 92 C9 BF EC 95 8E-EA 8F A4 EB 88 B4 EB 8E *................*

B7 EF BA B7 EF BF B8 C3-B8 3C 2F 72 6F 6F 74 3E *.........</root>*

00 00 00 00 7A *....z*

that cannot possibly appear in any of the documents; for example, a series of four nulls (

) followed by the letter z

\0\0\0\0z

This example shows how to use this field terminator for the xTable

sample table. To create this sample table, use the following

CREATE TABLE

statement:

USE tempdb;

CREATE TABLE xTable (xCol xml);

The field terminator must be specified in the format file. Example D uses a non-XML format file named

Xmltable.fmt

that contains the following:

9.0

1 SQLBINARY 0 0 "\0\0\0\0z" 1 xCol ""

You can use this format file to bulk import XML documents into the xTable

table by using a bcp

command or a

BULK INSERT

INSERT ... SELECT * FROM OPENROWSET(BULK...)

statement.

This example uses the

Xmltable.fmt

format file in a

BULK INSERT

statement to import the contents of an XML data file named

Xmltable.dat

BULK INSERT xTable

FROM 'C:\Xmltable.dat'

WITH (FORMATFILE = 'C:\Xmltable.fmt');