How to Use the SOAP API With Attachments

By default, CopyStorm will use the Salesforce REST API for Attachments and other tables which contain BASE64 columns. REST is used by default because it minimizes the amount of memory needed by CopyStorm (See the technical details section at the end of this document is you are interested in the details).

If using the REST API is a problem then a simple CopyStorm configuration file change can force the SOAP API to be used instead. The primary need we have seen for this is when a local virus checker or network filter is blocking attachment downloads from Salesforce.

Make CopyStorm use the SOAP API for Attachments

Open the file [CopyStorm]/config/TableRuleRegistry.xml and change:

<TableRule name="Attachment" predicate="bodyLength&lt;800000000" />

to

<TableRule name="Attachment" predicate="bodyLength&lt;800000000" useRestAPI="false"/>

Alternatively, if you are using a custom directory of CopyStorm configuration files, simply add the following rule to your private TableRuleRegistry.xml file.

<TableRule name="Attachment" useRestAPI="false"/>

Technical Details

Since Salesforce SOAP API returns an entire attachment in a singleĀ  response, the amount of memory required to parse the response and send the attachment to a target database can be huge (for huge attachments). The Salesforce REST API can be used to avoid loading an entire attachment into memory. Here is why:

  • The REST API returns an attachment BODY as a hyperlink.
  • CopyStorm downloads the hyperlink to a temporary file (taking very little memory).
  • The temporary file is loaded directly into the target database without ever being loaded into CopyStorm memory. No all database support this feature but those that do minimize CopyStorm memory usage.