Help! CopyStorm Did Not Copy/Update a Record or Field.

This articles explores situations where CopyStorm appears to miss an update or insert of a record from Salesforce and the solutions to correct the CopyStorm database. The root causes tend to be one of the following:

  • Salesforce permission changes altered what columns are visible to the credentials used when running CopyStorm.
  • Salesforce permission changes altered what records are visible to the credentials used when running CopyStorm.
  • There is a problem altering the CopyStorm schema to match Salesforce.
  • The “Salesforce believes in Time Travel” problem.

Salesforce Permission Changes Altered What Columns Are Visible

When CopyStorm discovers a new column from the Salesforce metadata it automatically creates the new column in the database and assumes that it must be a new column in Salesforce.  Here is the scenario where changing Salesforce permissions make it look like CopyStorm missed updating column.

  • The Salesforce credentials used by CopyStorm do not have permission to see the column Contact.MyCustom__c.
  • On Monday the Contact table is backed up with CopyStorm.
  • On Tuesday morning an administrator changes permissions to allow the CopyStorm Salesforce account to see Contact.MyCustom__c
  • On Tuesday afternoon an incremental backup of the Contact table happens.
    • CopyStorm create the column Contact.MyCustom__c in the database.
    • The value for all modified records are backed up — including their value for Contact.MyCustom__c
  • On late Tuesday afternoon someone notices that not all records in CopyStorm have the correct value in the MyCustom__c record. All records with timestamps earlier than Tuesday afternoon seem to have the problem.

How do you fix this issue? There are two easy approaches:

  • Use CopyStorm/Medic’s Refresh Columns tool.
  • Run CopyStorm with the following options:
    • Enter the name of the problem table in the Tables to Copy box.
    • Enable the Force Updates option.
    • Enter 1/1/1990 in the Modified Since option.
    • Run a backup.

Salesforce Permission Changes Altered What Records Are Visible

Since CopyStorm can only backup Salesforce records it can see,  if a Salesforce permission change makes new records available they normally will not be picked up until their timestamp changes. Here is how to make CopyStorm rescan all records in a table and pick up missing records.

  • Enter the name of the problem table in the Tables to Copy box.
  • Enter 1/1/1990 in the Modified Since option.
  • Run a Copy
  • CopyStorm will rescan all records and insert any missing records into the database.

Problem Altering CopyStorm Schema to Match Salesforce

If a field is altered or added in Salesforce and CopyStorm finds it impossible to make the same change to the database then CopyStorm will record an error and refuse to backup data to the table until the error is resolved.

The quickest way to fix this problem is:

  • Enter the name of the problem table in the Tables to Copy box.
  • Click on the Force Rebuild option.
  • Run a backup and CopyStorm will drop the problem table, recreate it, and then repopulate it with data from Salesforce.

If the quick fix is too drastic for your case, the see the How to Fix ALTER COLUMN Errors article.

The “Salesforce Believes in Time Travel” Problem

First, this problem does not occur unless CopyStorm internal parameters are modified but it is included in this article for completeness (and because it is rare condition that few people know about). The end of this section describes how to make it possible for the “Time Travel” issue to occur but you may have to wait months for it to happen!

CopyStorm achieves fast incremental backup of Salesforce by using the timestamp on Salesforce records. Logically CopyStorm does the following each time it runs:

  • Read the most recent timestamp from the CopyStorm database and call it “T”
  • Run SOQL that looks like: SELECT * FROM TableName WHERE timestamp >= T

The “Time Travel” problem is that Salesforce DOES NOT always commit records in timestamp order. This means that a record whose timestamp is 12:02 may show up in Salesforce BEFORE a record with a timestamp of 12:01. This does happen often but when it does it is likely that any program doing incremental copies based on a timestamp may:

  • Save all records modified at timestamp 12:02 or earlier in the first run MISSING the one record modified at 12:01.
  • Start searching for new records using timestamp 12:02 in the second run and NEVER SEE the 12:01 record.

Fortunately it is possible to pretend that Salesforce always writes records in timestamp order by not processing records that have been modified in the past 2 minutes. How did we pick 2 minutes? We worked with a lot of high volume sites and also simulated massive Salesforce loads that reproduced the problem.

Occasionally a person evaluating CopyStorm will decide that it is skipping a record when performing an experiment like:

  • Backup all Contacts
  • In Salesforce, modify a Contact.
  • Immediately run CopyStorm. It will copy no records because the change is less than 2 minutes old.

When the evaluator emails support@capstorm.com we point them to the original time travel post and suggest that they run CopyStorm again. Of course, it picks up the changed record the second time.

If you are doing CopyStorm work on a sandbox and are willing to live with the “time travel” problem you can tell CopyStorm to pretend that a safe amount of time to wait is 1 second. Here is how:

  • Select the Salesforce Optimization secton of the CopyStorm Configuration Tab.
  • Click on the Edit... button next to Internal Parameters.
  • Set SFDC Commit Latency and Timestamp Latency to 1.