CopyStorm 8.37.2, 8.34.2

Release Date: 11-Mar-2017

This is a major feature and bug fix release.

Backup Salesforce Metadata

With a single click on the Include Metadata checkbox, CopyStorm will backup all Salesforce metadata and maintain versions as your metadata changes. Of course, CopyStorm/Restore can be used to restore meta data with surgical precision.

Boost Record Update Performance

In previous versions CopyStorm would UPDATE records by updating all fields (even if they had not changed in the target database). A new feature, Optimize Update SQL, forces CopyStorm to generate optimal SQL UPDATE statements at the cost of comparing data from Salesforce with the existing data in the target database. On all database this gives a small performance boost. On some databases (especially those containing tables with 100s of columns) we have seen a performance boost of up to 5x. The difference in performance is often related to how well a database engine caches prepared SQL statements when a SQL statement is large. On SQL/Server, for example, SQL statements larger than around 4000 characters appear to avoid the prepared statement cache because of a limit in the stored procedure sp_preparesql.

This feature can be enabled globally on the Advanced Tab or for individual tables using XML configuration files.

Boost Record Insert Performance for Sparse Tables

In previous versions CopyStorm would INSERT records by inserting all fields (even if they were null). A new feature, Optimize Insert SQL, forces CopyStorm to generate INSERT statements that exclude NULL columns. In our experience the performance different is minor unless a table is extremely sparsely populated.

Use a Single ALTER Statement When Adding Multiple Column in a Session

If multiple columns were added to Salesforce, previous releases would add each column to the target database using a single ALTER statement. This release adds all columns using a single ALTER statement. Both MySQL and recent versions of SQL/Server 2012 may see a significant performance boost for this operation. Why?

MySQL essentially rebuilds a table any time an ALTER statement is executed.
Recent versions of SQL/Server do a complete table scan when a variable length column is added.

Support the Salesforce REST API

Since we have seen cases where a few of the security gateways (we will not name names) corrupt the SOAP envelope returned by Salesforce, this release supports using the Salesforce REST API instead. In our experience the SOAP API tends to be a bit faster (probably because the XML parser is faster than the JSON parser) so we do not recommend enabling the REST API unless SOAP is causing issues.

Add Table Name to Recorded Exceptions

If an exception occurs when working on a specific table CopyStorm will now record the table name as a separate column in the CopyForceJobException table.

Display Connection Info If the Capstorm License Server is Inaccessible

If the computer running CopyStorm needs a license and cannot access the Capstorm license server, htttp://license.capstorm.com, the application will display a dialog which contains the information needed by Capstorm support to create a license manually.

Normalize System Exit Codes

CopyStorm will return a 0 exit code there were no problems, 1 if there were issues that need investigation but may be transitory, 2 if there were severe errors that require investigation.

Fix a Rare NPE When Processing Knowledge Articles

If the provided Salesforce credentials could see Knowledge tables BUT had no permission the query them for data then CopyStorm would throw a null pointer exception.

Switch to Bulk API Mode Even if a Bulk LIMIT is Not Specified

In the previous version, CopyStorm would not use the Bulk API unless the Bulk Limit parameter was specified. In this update the Bulk Limit will default to the normal row limit if not specified.

Exclude Case Comment Fields That Fail to Read

A Salesforce API bug prevents two CaseComment fields from being read directly — CreatorFullPhotoURL and CreatorSmallPhotoURL. When a SOQL statement attempts to read these fields an INVALID_REFERENCE_FIELD exception occurs. This version of CopyStorm excludes these fields since their values are also found in the User table.

Fix a Bug When a Secondary Reader Thread Fails

A “Secondary Reader Thread” occurs in CopyStorm when a table is too wide to be read using a single SOQL statement. A bug was found in CopyStorm when a secondary (or tirtuary) thread encountered a Salesforce QUERY_TIMEOUT that lead to CopyStorm hanging rather than restarting the entire query.

Support Backup of Tables without Timestamp Fields

There are a handful of system tables in Salesforce that do not contain any sort of timestamp column. In this release CopyStorm will backup these tables using an ordering based on the table’s ID column.

Support Backup of the ChatterMessage Table

The ChatterMessage has a non-standard timestamp column named “SentDate” that was not recognized by previous CopyStorm versions.

Change the Default SQL Typed Used for Salesforce DATE to DATE

In SQL/Server the SQL type use for Salesforce columns has been DATETIME. In the release newly created DATE columns CopyStorm will use the DATE type in SQL/Server.

Code Around a Problem with Salesforce Statistic Tables

Salesforce will often report view ridiculous statistic values of 1.0e-100 in various tables whose names end with _stat. Such a small value is obviously incorrect and can cause underflow problems in certain databases. CopyStorm codes around the problem by recognizing ridiculously small values and changing them to zero. Until astrophysics or chemistry starts using Salesforce as a primary data platform it is doubtful that values smaller than the radius of a proton will be meaningful in Salesforce (e.g. 0.84e-15 meters). Note that this sort of value has also been seen at one customer site when a formula was mistakenly dividing by zero.

Detect When the Bulk API Rejects a Request

The previous release was not catching the case where the Bulk API refused a request with a NotProcessed response (a NotProcessed request will never be executed by the Bulk API).

Fix a Bug in Database Based History Records

If the backup of a table encountered recoverable timeouts the total number of records copied in the database resident history table would only reflect the number of records copied since the most recent timeout.