CopyStorm 9.47.2

Release Date: 29-Jan-2020

This is a  feature and  bug fix release. As always, existing CopyStorm configuration files are backwards compatible.

NOTE: Please Upgrade CopyStorm/Medic when you start using this release. A performance upgrade for detecting and processing deleted records requires that both CopyStorm and CopyStorm/Medic be upgraded concurrently. If you fail to do this, CopyStorm may not remove records hard-deleted by Salesforce when detected by CopyStorm/Medic.

Patch Levels

Supercharge Backup Rate for Attachments and Other Tables with Base64 Columns

The rate at which any table with a base64 column (Attachment, ContentVersion, etc.) is backed up is increased significantly in this update by adding a feature to concurrently fetch multiple record bodies (base64 columns).  In all tests with clients, the update was rate limited by:

• the rate of the internet connection (the update easily pegged out a 1GB line)
• the rate at which the database would accept data

In our own labs we see between a 5x and 12x performance boost.  By default, this version of CopyStorm will process 5 base64 columns concurrently. However, this value can be changed with the Max Base64 Reader Threads option on the Internal Parameters dialog.  Be careful about changing this parameter to  20 or greater. We have some evidence that Salesforce may interpret this as a DoS attack and temporarily suspend your IP.

Greatly Improve the Performance of Detecting Deleted Records

The amount of time CopyStorm takes to deleted records based on the “Delete Older Than” option on the Advanced Tab should decrease dramatically the second time this version of CopyStorm runs. This release uses a new bit of meta data stored in CopyForceTable to minimize the amount of records that must be scanned.

If you fail to upgrade CopyStorm/Medic as part of this CopyStorm upgrade, you may need to turn this feature off until CopyStorm/Medic is upgraded. The feature can be turned off by unchecking the “Optimize Deleted Record Scan” option on the Internal Parameters dialog.

Improve Detection of a Salesfore SELECT .. LIMIT “Feature“

For ages CopyStorm has known that “SELECT … FROM table WHERE … LIMIT 10000” will on rare occasions return less than 10,000 records even when more than 10,000 records match the criteria. In recent history we have some evidence that this “feature” is becoming more common on older Salesforce instances. Our suspicion is that Salesforce is trying to save a few $$ by putting older data on slower storage and internal query timeouts are preventing all 10,000 records from being found. Independent of the root cause of the now more frequent “feature“,  we have coded around it.

Add SOQL and SQL Query Tools

A new top level menu item , Tools, contains dialogs for running SOQL and SQL queries. Why were these added? Enough customers asked for these features. (thank you!)

Enhance the Test Salesforce Connection Dialog

The Test Salesforce Connection dialog displays the status of all Salesforce permissions which are recommended when backing  up a Salesforce instance and explains why each permission is recommended.

Add Support for SFDX Credentials on the Command Line

We use scratch orgs a lot in our automated build/test system and use a command line feature to force a batch CopyStorm job to use sfdx user credentials. We should have documented this ages ago for others. To force authentication to a scratch org from a command line:

• CopyStorm -sftype sfdx -sfuser sfdxAlias

Add Support for Completely Ignoring Fields in Salesforce

A new xml configuration directive has been added to support completely ignoring a Salesforce field when doing a backup. A completely ignored field will not appear in the target database or in CopyForceTableField.  See this article (at the end) for details.

Recover from REST Errors Mid Stream

In past release, if the REST API failed then CopyStorm would automatically revert to SOAP for the rest of the job. In this version, CopyStorm will revert to SOAP to process the record causing the issue in REST and then immediately revert back to REST. This feature ensures that concurrent attachment backup will only be paused for a single batch of records when the REST API throws a 404 or 500 error (this is a known REST issue for certain objects).

The Archive Subsystem is Disabled for Oracle Versions Prior to 12

The Archive subsystem never worked on pre-12 versions of Oracle — this update makes the failure graceful.  The core issue is that the Archive subsystem uses Oracle IDENTITY columns and these were not introduced until Oracle 12 ( this is probably not an issue for your shop. Oracle 11g extended support ends in December 2020. Orace 12 first was release in 2013)

Add Feature to Force Table Truncation from the Command Line

A new command line switch, -forceTruncate, will force all tables in a CopyStorm config file to be truncated before being backed up.  The effect is exactly the same as if the force truncate option was checked on the Advanced Tab.