CopyStorm Configuration Files

Many sites will decide to backup their Salesforce using multiple CopyStorm jobs. Why?

• A site will want the data in specific tables in the CopyStorm database to be within minutes of the data in Salesforce. These tables in CopyStorm are frequently used for analytics or operational reporting.
• Some tables in Salesforce simply take longer to backup and are not important to keep within minutes of Salesforce. Common culprits include Attachment, ContentVersion, and Knowledge Article version tables (each of these tables contain base64 columns and take a single API call to read a record).

If you are not content with a single CopyStorm job then this article describes a few best practices for using multiple CopyStorm jobs.

A Common Setup Strategy

A common setup strategy involves dividing the tables in Salesforce into three separate buckets.

1. Tables to backup frequently.
2. Tables that are expensive to backup and can be backed up less frequently.
3. Metadata.

For this setup, we create three separate CopyStorm configuration files and schedule them to run a different times and frequencies.

Frequent Configuration File

This configuration file is will be run frequently and backs up tables we want to keep in close agreement with Salesforce. Rather than specify what tables are in the frequent category, we specify the tables to exclude. Many sites will run this backup every 15 to 30 minutes.

Less Frequent Configuration File

This configuration file will be run infrequently and backs up tables that do not need to be kept within a few minutes of Salesforce. Use the “Include List” CopyStorm feature to specify the tables to back up. Many sites run this backup every few hours.

Metadata Configuration File

This configuration file will be run infrequently and backs up just Salesforce metadata. Use the “Include List” CopyStorm feature to tell CopyStorm to backup up just a single table — in this case the table specified is not in Salesforce.

Next Steps

Once multiple CopyStorm configuration files have been created, jobs need to be created and scheduled.

• Store the CopyStorm configuration files using Capstorm’s Directory Structure Best Practices.
• Create a script for running each job using the CopyStorm Batch File tab.
• Schedule each job.

Advanced Strategies

There is no functional limit to the number of CopyStorm jobs you can use to backup Salesforce and some sites will use four or more jobs — usually to support keeping key CopyStorm tables within minutes of Salesforce. The choice is really yours.

Here are few suggestions when running many jobs.

• Try to avoid backing up the same table in multiple jobs. Though this will not damage the integrity of the backup it will often generate annoying messages.
• Consider using a “super scheduler” tool like CopyStorm/Director to manage scheduling and logs. In general a super-scheduler make managing a lot of backup jobs and their logs much simpler.
• Be aware that a single Salesforce Login Name can have at most 20 concurrent active queries. If you have more than 20, Salesforce may silently kill one of your sessions. Though CopyStorm will recover from this unfriendly Salesforce action automatically,  many applications will not behave so nicely.