Sample Client Application Using Java. salesforce Bulk API
विज्ञापन
विज्ञापन
APPENDIX A Sample Client Application Using Java
Use the code sample in this section to create a test client application that inserts a number of account records using the REST-based
Bulk API.
In addition to the step-by-step instructions that follow, the end of this section provides the complete code for you, to make copying and pasting easier.
Note: Before you begin building an integration or other client application:
•
Install your development platform according to its product documentation.
•
Read through all the steps before creating the test client application. You may also wish to review the rest of this document to familiarize yourself with terms and concepts.
Set Up Your Client Application
The Bulk API uses HTTP GET and HTTP POST methods to send and receive XML content, so it's very simple to build clients in the language of your choice. This section uses a Java sample and the Salesforce Web Service Connector (WSC) toolkit provided by Salesforce to simplify development. WSC is a high-performing web service client stack implemented using a streaming parser. The toolkit has built-in support for the basic operations and objects used in the Bulk API.
Review the library here: https://github.com/forcedotcom/wsc
To download the Salesforce WSC toolkit:
1. Browse to http://mvnrepository.com/artifact/com.force.api/force-wsc
2. Click the “Available versions” link that matches the API version you are using.
3. Click Download (JAR) and save the file to a local directory.
The Bulk API does not provide a login operation, so you must use the SOAP API to login.
To download the partner WSDL and compile it to Java classes with the WSC toolkit:
1. Log in to your Developer Edition Salesforce account. You must log in as an administrator or as a user who has the “Modify All Data” permission. Logins are checked to ensure they are from a known IP address. For more information, see “Setting Login Restrictions” in the Salesforce online help.
2. From Setup, click Develop > API.
3. Right-click Partner WSDL to display your browser's save options, and save the partner WSDL to a local directory. For information about the partner WSDL, see Using the Partner WSDL .
4. Compile the partner API code from the WSDL using the WSC compile tool: java -classpath
pathToJar
.\wsdlGenFiles.jar
\wsc.jar com.sforce.ws.tools.wsdlc
pathToWSDL\wsdlFilename
64
Sample Client Application Using Java Walk Through the Sample Code
For example, if wsc.jar
is installed in
C:\salesforce\wsc
, and the partner WSDL is saved to
C:\salesforce\wsdl\partner
: java -classpath C:\salesforce\wsc\wsc.jar
com.sforce.ws.tools.wsdlc
C:\salesforce\wsdl\partner\partner.wsdl
.\partner.jar
wsc.jar
and the generated partner.jar
are the only libraries needed in the classpath for the code examples in the following sections.
Walk Through the Sample Code
Once you have set up your client, you can begin building client applications that use the Bulk API. Use the following sample to create a client application. Each section steps through part of the code. The complete sample is included at the end.
The following code sets up the packages and classes in the WSC toolkit and the code generated from the partner WSDL: import java.io.*; import java.util.*; import com.sforce.async.*; import com.sforce.soap.partner.PartnerConnection; import com.sforce.ws.ConnectionException; import com.sforce.ws.ConnectorConfig;
Set Up the
main()
Method
This code sets up the main()
method for the class. It calls the runSample()
method, which encompasses the processing logic for the sample. We'll look at the methods called in runSample()
in subsequent sections.
public static void main(String[] args) throws AsyncApiException, ConnectionException, IOException {
BulkExample example = new BulkExample();
// Replace arguments below with your credentials and test file name
// The first parameter indicates that we are loading Account records example.runSample("Account", "[email protected]", "myPassword", "mySampleData.csv");
}
/**
* Creates a Bulk API job and uploads batches for a CSV file.
*/ public void runSample(String sobjectType, String userName,
String password, String sampleFileName) throws AsyncApiException, ConnectionException, IOException {
BulkConnection connection = getBulkConnection(userName, password);
JobInfo job = createJob(sobjectType, connection);
List<BatchInfo> batchInfoList = createBatchesFromCSVFile(connection, job, sampleFileName); closeJob(connection, job.getId()); awaitCompletion(connection, job, batchInfoList);
65
Sample Client Application Using Java Walk Through the Sample Code
} checkResults(connection, job, batchInfoList);
Login and Configure BulkConnection
The following code logs in using a partner connection (PartnerConnection) and then reuses the session to create a Bulk API connection
(BulkConnection).
/**
* Create the BulkConnection used to call Bulk API operations.
*/ private BulkConnection getBulkConnection(String userName, String password) throws ConnectionException, AsyncApiException {
ConnectorConfig partnerConfig = new ConnectorConfig(); partnerConfig.setUsername(userName); partnerConfig.setPassword(password); partnerConfig.setAuthEndpoint("https://login.salesforce.com/services/Soap/u/34.0");
// Creating the connection automatically handles login and stores
// the session in partnerConfig new PartnerConnection(partnerConfig);
// When PartnerConnection is instantiated, a login is implicitly
// executed and, if successful,
// a valid session is stored in the ConnectorConfig instance.
// Use this key to initialize a BulkConnection:
ConnectorConfig config = new ConnectorConfig(); config.setSessionId(partnerConfig.getSessionId());
// The endpoint for the Bulk API service is the same as for the normal
// SOAP uri until the /Soap/ part. From here it's '/async/versionNumber'
String soapEndpoint = partnerConfig.getServiceEndpoint();
String apiVersion = "34.0";
String restEndpoint = soapEndpoint.substring(0, soapEndpoint.indexOf("Soap/"))
+ "async/" + apiVersion; config.setRestEndpoint(restEndpoint);
// This should only be false when doing debugging.
config.setCompression(true);
// Set this to true to see HTTP requests and responses on stdout config.setTraceMessage(false);
BulkConnection connection = new BulkConnection(config); return connection;
}
This BulkConnection instance is the base for using the Bulk API. The instance can be reused for the rest of the application life span.
Create a New Job
After creating the connection, create a new job. Data is always processed in the context of a job. The job specifies the details about the data being processed: what operation is being executed (insert, update, upsert, or delete) and the object type. The following code creates a new insert job on the Account object.
/**
66
Sample Client Application Using Java Walk Through the Sample Code
* Create a new job using the Bulk API.
*
* @param sobjectType
* The object type being loaded, such as "Account"
* @param connection
* BulkConnection used to create the new job.
* @return The JobInfo for the new job.
* @throws AsyncApiException
*/ private JobInfo createJob(String sobjectType, BulkConnection connection) throws AsyncApiException {
JobInfo job = new JobInfo(); job.setObject(sobjectType); job.setOperation(OperationEnum.insert); job.setContentType(ContentType.CSV); job = connection.createJob(job);
System.out.println(job); return job;
}
When a job is created, it's in the
Open
state. In this state new batches can be added to the job. Once a job is
Closed
, batches can no longer be added.
Add Batches to the Job
Data is processed in a series of batch requests. Each request is an HTTP POST containing the data set in XML format in the body. Your client application determines how many batches are used to process the whole data set as long as the batch size and total number of batches per day are within the limits specified in
The processing of each batch comes with an overhead. Batch sizes should be large enough to minimize the overhead processing cost and small enough to be easily handled and transferred. Batch sizes between 1,000 and 10,000 records are considered reasonable.
The following code splits a CSV file into smaller batch files and uploads them to Salesforce.
/**
* Create and upload batches using a CSV file.
* The file into the appropriate size batch files.
*
* @param connection
* Connection to use for creating batches
* @param jobInfo
* Job associated with new batches
* @param csvFileName
* The source file for batch data
*/ private List<BatchInfo> createBatchesFromCSVFile(BulkConnection connection,
JobInfo jobInfo, String csvFileName) throws IOException, AsyncApiException {
List<BatchInfo> batchInfos = new ArrayList<BatchInfo>();
BufferedReader rdr = new BufferedReader( new InputStreamReader(new FileInputStream(csvFileName))
);
// read the CSV header row byte[] headerBytes = (rdr.readLine() + "\n").getBytes("UTF-8");
67
Sample Client Application Using Java Walk Through the Sample Code int headerBytesLength = headerBytes.length;
File tmpFile = File.createTempFile("bulkAPIInsert", ".csv");
// Split the CSV file into multiple batches try {
FileOutputStream tmpOut = new FileOutputStream(tmpFile); int maxBytesPerBatch = 10000000; // 10 million bytes per batch int maxRowsPerBatch = 10000; // 10 thousand rows per batch int currentBytes = 0; int currentLines = 0;
String nextLine; while ((nextLine = rdr.readLine()) != null) { byte[] bytes = (nextLine + "\n").getBytes("UTF-8");
// Create a new batch when our batch size limit is reached if (currentBytes + bytes.length > maxBytesPerBatch
|| currentLines > maxRowsPerBatch) { createBatch(tmpOut, tmpFile, batchInfos, connection, jobInfo); currentBytes = 0; currentLines = 0;
} if (currentBytes == 0) { tmpOut = new FileOutputStream(tmpFile); tmpOut.write(headerBytes); currentBytes = headerBytesLength; currentLines = 1;
} tmpOut.write(bytes); currentBytes += bytes.length; currentLines++;
}
// Finished processing all rows
// Create a final batch for any remaining data if (currentLines > 1) { createBatch(tmpOut, tmpFile, batchInfos, connection, jobInfo);
}
} finally { tmpFile.delete();
} return batchInfos;
}
/**
* Create a batch by uploading the contents of the file.
* This closes the output stream.
*
* @param tmpOut
* The output stream used to write the CSV data for a single batch.
* @param tmpFile
* The file associated with the above stream.
* @param batchInfos
* The batch info for the newly created batch is added to this list.
* @param connection
* The BulkConnection used to create the new batch.
* @param jobInfo
68
Sample Client Application Using Java Walk Through the Sample Code
*
*/
The JobInfo associated with the new batch.
private void createBatch(FileOutputStream tmpOut, File tmpFile,
List<BatchInfo> batchInfos, BulkConnection connection, JobInfo jobInfo) throws IOException, AsyncApiException { tmpOut.flush(); tmpOut.close();
FileInputStream tmpInputStream = new FileInputStream(tmpFile); try {
BatchInfo batchInfo = connection.createBatchFromStream(jobInfo, tmpInputStream);
System.out.println(batchInfo); batchInfos.add(batchInfo);
} finally { tmpInputStream.close();
}
}
Once the server receives a batch it's immediately queued for processing. Any errors in formatting aren't reported when sending the batch. These errors are reported in the result data when the batch is processed.
Tip:
To import binary attachments, use the following methods. Specify the CSV or XML content for the batch in the batchContent
parameter, or include request.txt
in the attached files and pass null
to the batchContent parameter. These methods are contained within the com.async.BulkConnection
class:
•
createBatchFromDir()
•
createBatchWithFileAttachments()
•
createBatchWithInputStreamAttachments()
•
createBatchFromZipStream()
Close the Job
After all batches have been added to a job, close the job. Closing the job ensures that processing of all batches can finish.
private void closeJob(BulkConnection connection, String jobId) throws AsyncApiException {
JobInfo job = new JobInfo(); job.setId(jobId); job.setState(JobStateEnum.Closed); connection.updateJob(job);
}
Check Status on Batches
Batches are processed in the background. A batch may take some time to complete depending on the size of the data set. During processing, the status of all batches can be retrieved and checked to see when they have completed.
/**
* Wait for a job to complete by polling the Bulk API.
69
Sample Client Application Using Java Walk Through the Sample Code
*
* @param connection
*
* @param job
BulkConnection used to check results.
* The job awaiting completion.
* @param batchInfoList
* List of batches for this job.
* @throws AsyncApiException
*/ private void awaitCompletion(BulkConnection connection, JobInfo job,
List<BatchInfo> batchInfoList) throws AsyncApiException { long sleepTime = 0L;
Set<String> incomplete = new HashSet<String>(); for (BatchInfo bi : batchInfoList) { incomplete.add(bi.getId());
} while (!incomplete.isEmpty()) { try {
Thread.sleep(sleepTime);
} catch (InterruptedException e) {}
System.out.println("Awaiting results..." + incomplete.size()); sleepTime = 10000L;
BatchInfo[] statusList = connection.getBatchInfoList(job.getId()).getBatchInfo(); for (BatchInfo b : statusList) { if (b.getState() == BatchStateEnum.Completed
|| b.getState() == BatchStateEnum.Failed) { if (incomplete.remove(b.getId())) {
System.out.println("BATCH STATUS:\n" + b);
}
}
}
}
}
A batch is done when it's either failed or completed. This code loops infinitely until all the batches for the job have either failed or completed.
Get Results For a Job
After all batches have completed, the results of each batch can be retrieved. Results should be retrieved whether the batch succeeded or failed, or even when the job was aborted, because only the result sets indicate the status of individual records. To properly pair a result with its corresponding record, the code must not lose track of how the batches correspond to the original data set. This can be achieved by keeping the original list of batches from when they were created and using this list to retrieve results, as shown in the following example:
/**
* Gets the results of the operation and checks for errors.
*/ private void checkResults(BulkConnection connection, JobInfo job,
List<BatchInfo> batchInfoList)
70
Sample Client Application Using Java Walk Through the Sample Code throws AsyncApiException, IOException {
// batchInfoList was populated when batches were created and submitted for (BatchInfo b : batchInfoList) {
CSVReader rdr = new CSVReader(connection.getBatchResultStream(job.getId(), b.getId()));
List<String> resultHeader = rdr.nextRecord(); int resultCols = resultHeader.size();
List<String> row; while ((row = rdr.nextRecord()) != null) {
Map<String, String> resultInfo = new HashMap<String, String>(); for (int i = 0; i < resultCols; i++) { resultInfo.put(resultHeader.get(i), row.get(i));
} boolean success = Boolean.valueOf(resultInfo.get("Success")); boolean created = Boolean.valueOf(resultInfo.get("Created"));
String id = resultInfo.get("Id");
String error = resultInfo.get("Error"); if (success && created) {
System.out.println("Created row with id " + id);
} else if (!success) {
System.out.println("Failed with error: " + error);
}
}
}
}
This code retrieves the results for each record and reports whether the operation succeeded or failed. If an error occurred for a record, the code prints out the error.
Complete Quick Start Sample
Now that you're more familiar with jobs and batches, you can copy and paste the entire quick start sample and use it: import java.io.*; import java.util.*; import com.sforce.async.*; import com.sforce.soap.partner.PartnerConnection; import com.sforce.ws.ConnectionException; import com.sforce.ws.ConnectorConfig; public class BulkExample { public static void main(String[] args) throws AsyncApiException, ConnectionException, IOException {
BulkExample example = new BulkExample();
// Replace arguments below with your credentials and test file name
// The first parameter indicates that we are loading Account records example.runSample("Account", "[email protected]", "myPassword", "mySampleData.csv");
71
Sample Client Application Using Java Walk Through the Sample Code
}
/**
* Creates a Bulk API job and uploads batches for a CSV file.
*/ public void runSample(String sobjectType, String userName,
String password, String sampleFileName) throws AsyncApiException, ConnectionException, IOException {
BulkConnection connection = getBulkConnection(userName, password);
JobInfo job = createJob(sobjectType, connection);
List<BatchInfo> batchInfoList = createBatchesFromCSVFile(connection, job, sampleFileName); closeJob(connection, job.getId()); awaitCompletion(connection, job, batchInfoList); checkResults(connection, job, batchInfoList);
}
/**
* Gets the results of the operation and checks for errors.
*/ private void checkResults(BulkConnection connection, JobInfo job,
List<BatchInfo> batchInfoList) throws AsyncApiException, IOException {
// batchInfoList was populated when batches were created and submitted for (BatchInfo b : batchInfoList) {
CSVReader rdr = new CSVReader(connection.getBatchResultStream(job.getId(), b.getId()));
List<String> resultHeader = rdr.nextRecord(); int resultCols = resultHeader.size();
List<String> row; while ((row = rdr.nextRecord()) != null) {
Map<String, String> resultInfo = new HashMap<String, String>(); for (int i = 0; i < resultCols; i++) { resultInfo.put(resultHeader.get(i), row.get(i));
} boolean success = Boolean.valueOf(resultInfo.get("Success")); boolean created = Boolean.valueOf(resultInfo.get("Created"));
String id = resultInfo.get("Id");
String error = resultInfo.get("Error"); if (success && created) {
System.out.println("Created row with id " + id);
} else if (!success) {
System.out.println("Failed with error: " + error);
}
}
}
} private void closeJob(BulkConnection connection, String jobId)
72
Sample Client Application Using Java
} throws AsyncApiException {
JobInfo job = new JobInfo(); job.setId(jobId); job.setState(JobStateEnum.Closed); connection.updateJob(job);
Walk Through the Sample Code
/**
* Wait for a job to complete by polling the Bulk API.
*
* @param connection
*
* @param job
BulkConnection used to check results.
* The job awaiting completion.
* @param batchInfoList
* List of batches for this job.
* @throws AsyncApiException
*/ private void awaitCompletion(BulkConnection connection, JobInfo job,
List<BatchInfo> batchInfoList) throws AsyncApiException { long sleepTime = 0L;
Set<String> incomplete = new HashSet<String>(); for (BatchInfo bi : batchInfoList) { incomplete.add(bi.getId());
} while (!incomplete.isEmpty()) { try {
Thread.sleep(sleepTime);
} catch (InterruptedException e) {}
System.out.println("Awaiting results..." + incomplete.size()); sleepTime = 10000L;
BatchInfo[] statusList = connection.getBatchInfoList(job.getId()).getBatchInfo(); for (BatchInfo b : statusList) { if (b.getState() == BatchStateEnum.Completed
|| b.getState() == BatchStateEnum.Failed) { if (incomplete.remove(b.getId())) {
System.out.println("BATCH STATUS:\n" + b);
}
}
}
}
}
/**
* Create a new job using the Bulk API.
*
* @param sobjectType
* The object type being loaded, such as "Account"
73
Sample Client Application Using Java Walk Through the Sample Code
* @param connection
* BulkConnection used to create the new job.
* @return The JobInfo for the new job.
* @throws AsyncApiException
*/ private JobInfo createJob(String sobjectType, BulkConnection connection) throws AsyncApiException {
JobInfo job = new JobInfo(); job.setObject(sobjectType); job.setOperation(OperationEnum.insert); job.setContentType(ContentType.CSV); job = connection.createJob(job);
System.out.println(job); return job;
}
/**
* Create the BulkConnection used to call Bulk API operations.
*/ private BulkConnection getBulkConnection(String userName, String password) throws ConnectionException, AsyncApiException {
ConnectorConfig partnerConfig = new ConnectorConfig(); partnerConfig.setUsername(userName); partnerConfig.setPassword(password); partnerConfig.setAuthEndpoint("https://login.salesforce.com/services/Soap/u/34.0");
// Creating the connection automatically handles login and stores
// the session in partnerConfig new PartnerConnection(partnerConfig);
// When PartnerConnection is instantiated, a login is implicitly
// executed and, if successful,
// a valid session is stored in the ConnectorConfig instance.
// Use this key to initialize a BulkConnection:
ConnectorConfig config = new ConnectorConfig(); config.setSessionId(partnerConfig.getSessionId());
// The endpoint for the Bulk API service is the same as for the normal
// SOAP uri until the /Soap/ part. From here it's '/async/versionNumber'
String soapEndpoint = partnerConfig.getServiceEndpoint();
String apiVersion = "34.0";
String restEndpoint = soapEndpoint.substring(0, soapEndpoint.indexOf("Soap/"))
+ "async/" + apiVersion; config.setRestEndpoint(restEndpoint);
// This should only be false when doing debugging.
config.setCompression(true);
// Set this to true to see HTTP requests and responses on stdout config.setTraceMessage(false);
BulkConnection connection = new BulkConnection(config); return connection;
}
74
Sample Client Application Using Java Walk Through the Sample Code
/**
* Create and upload batches using a CSV file.
* The file into the appropriate size batch files.
*
* @param connection
* Connection to use for creating batches
* @param jobInfo
* Job associated with new batches
* @param csvFileName
* The source file for batch data
*/ private List<BatchInfo> createBatchesFromCSVFile(BulkConnection connection,
JobInfo jobInfo, String csvFileName) throws IOException, AsyncApiException {
List<BatchInfo> batchInfos = new ArrayList<BatchInfo>();
BufferedReader rdr = new BufferedReader( new InputStreamReader(new FileInputStream(csvFileName))
);
// read the CSV header row byte[] headerBytes = (rdr.readLine() + "\n").getBytes("UTF-8"); int headerBytesLength = headerBytes.length;
File tmpFile = File.createTempFile("bulkAPIInsert", ".csv");
// Split the CSV file into multiple batches try {
FileOutputStream tmpOut = new FileOutputStream(tmpFile); int maxBytesPerBatch = 10000000; // 10 million bytes per batch int maxRowsPerBatch = 10000; // 10 thousand rows per batch int currentBytes = 0; int currentLines = 0;
String nextLine; while ((nextLine = rdr.readLine()) != null) { byte[] bytes = (nextLine + "\n").getBytes("UTF-8");
// Create a new batch when our batch size limit is reached if (currentBytes + bytes.length > maxBytesPerBatch
|| currentLines > maxRowsPerBatch) { createBatch(tmpOut, tmpFile, batchInfos, connection, jobInfo); currentBytes = 0; currentLines = 0;
} if (currentBytes == 0) { tmpOut = new FileOutputStream(tmpFile); tmpOut.write(headerBytes); currentBytes = headerBytesLength; currentLines = 1;
} tmpOut.write(bytes); currentBytes += bytes.length; currentLines++;
}
// Finished processing all rows
// Create a final batch for any remaining data if (currentLines > 1) { createBatch(tmpOut, tmpFile, batchInfos, connection, jobInfo);
75
Sample Client Application Using Java Walk Through the Sample Code
}
} finally { tmpFile.delete();
} return batchInfos;
}
/**
* Create a batch by uploading the contents of the file.
* This closes the output stream.
*
* @param tmpOut
* The output stream used to write the CSV data for a single batch.
* @param tmpFile
* The file associated with the above stream.
* @param batchInfos
* The batch info for the newly created batch is added to this list.
* @param connection
* The BulkConnection used to create the new batch.
* @param jobInfo
*
*/
The JobInfo associated with the new batch.
private void createBatch(FileOutputStream tmpOut, File tmpFile,
List<BatchInfo> batchInfos, BulkConnection connection, JobInfo jobInfo) throws IOException, AsyncApiException { tmpOut.flush(); tmpOut.close();
FileInputStream tmpInputStream = new FileInputStream(tmpFile); try {
BatchInfo batchInfo = connection.createBatchFromStream(jobInfo, tmpInputStream);
System.out.println(batchInfo); batchInfos.add(batchInfo);
} finally { tmpInputStream.close();
}
}
}
76
डाउनलोड
सार्वजनिक लिंक अपडेट किया गया
आपकी चैट का सार्वजनिक लिंक अपडेट कर दिया गया है. You may view and manage your chat links after registration here.
दस्तावेज़ सार्वजनिक लिंक
विज्ञापन
