CapStorm applications connect to Salesforce using several supported OAuth login types. This section covers the available options:

• JWT

• Client Credentials

• Username-Password

Each flow offers a different approach to authentication depending on how your Salesforce org is configured.

CaseMilestone is an example of a Salesforce table that contains formula columns whose values may change even when other fields of a CaseMilestone record do not change. In fact, the CaseMilestone table contains both types of Salesforce formula columns difficult to include in an incremental backup:

1. A formula field whose value changes when the value in another record (in the same table or a different table) changes.

2. A formula field whose value is computed using the current date or time.

To identify Salesforce formulas in a Salesforce standard table look for non-reference field columns where the API does not have update access. Knowing whether a formula field value will change even when the corresponding fields in the record do not change is a bit more difficult.

Fixing The Problem

The solution to this problem is to periodically run the CopyStorm/Medic Refresh Columns tool to bring the problem formula columns back in sync with Salesforce.

The following table lists Salesforce defined formula fields that are known to have the issue described above:

To connect to Salesforce using the OAUTH CLIENT CREDENTIALS flow, CopyStorm can be configured with:

1. The OAuth Client ID

2. The OAuth Client Secret

OAuth Client Credentials can be enabled in Salesforce by following this tutorial.

OAuth Client Credential logins can be enabled in CopyStorm by navigating to the Advanced Parameters dialog on the Main tab:

After selecting the OAuth 2.0 Client Credentials login type, provide the credentials needed to authenticate with Salesforce:

To connect to Salesforce using the OAUTH CLIENT CREDENTIALS flow, CopyStorm can be configured with:

1. OAuth Client ID

2. OAuth Client Secret

3. Username

4. Password

OAuth Client Credentials can be enabled in Salesforce by following .

OAuth Client Credential logins can be enabled in CopyStorm by navigating to the Advanced Parameters dialog on the Main tab:

After selecting the OAuth 2.0 Username-Password login type, provide the credentials needed to authenticate with Salesforce:

When connecting to S3, by default, CapStorm applications will:

1. Read Permanent Credentials provided using the AWS Provider Chain

2. Use these credentials to request temporary session credentials via the AWS STS (Security Token Service).

3. Periodically refresh the temporary credentials for continued access.

This approach minimizes the use of permanent credentials by keeping them off the wire during ongoing operations.

Note: For CopyStorm versions 10.56.6 and higher, Java memory parameters can be provided without modifying the CopyStorm start script. For more details, please see .

CopyStorm uses the default settings for Java heap space. Most of the times this works well, but if you are seeing an Out of Memory error referencing Java Heap Space you may need to increase the memory allowance for CopyStorm. Settings can be adjusted as described in the “CopyStorm.bat” file, but the exact settings will depend on the size needed for the data in your Salesforce instance.

The most common reason to adjust the default memory settings is that a large content item (usually an Attachment, Document, or ContentVersion) is too large to fit in available CopyStorm memory. CopyStorm requires 3 to 4 times the size of the largest content item. Here are the steps to find the largest content items in your Salesforce:

In the management menu found by clicking on your name in the upper right-hand corner, go to:

Setup -> Data Management -> Storage Usage -> Overall Top Files by Storage Utilization

Though CopyStorm keeps a data in sync with Salesforce, there are a few cases where the data can become out of sync:

• A new column is created and populated in Salesforce, but the Salesforce credentials used by CopyStorm cannot see the column.

  • When permissions are changed and CopyStorm can see the column, any data populated prior to CopyStorm having visibility will be null and the data will need to be propagated.

• A column contains a formula whose value changes without the timestamp on the row changing.

By default, CopyStorm will use the Salesforce REST API for Attachments and other tables which contain BASE64 columns. REST is used by default because it minimizes the amount of memory needed by CopyStorm (See the technical details section at the end of this document is you are interested in the details).

If using the REST API is a problem then a simple CopyStorm configuration file change can force the SOAP API to be used instead. The primary need we have seen for this is when a local virus checker or network filter is blocking attachment downloads from Salesforce.

Make CopyStorm use the SOAP API for Attachments

Open the file [CopyStorm]/config/TableRuleRegistry.xml and change:

<TableRule name="Attachment" predicate="bodyLength&lt;800000000" />

to

<TableRule name="Attachment" predicate="bodyLength&lt;800000000" useRestAPI="false"/>

Alternatively, if you are using a custom directory of CopyStorm configuration files, simply add the following rule to your private TableRuleRegistry.xml file.

<TableRule name="Attachment" useRestAPI="false"/>

Since Salesforce SOAP API returns an entire attachment in a single response, the amount of memory required to parse the response and send the attachment to a target database can be huge (for huge attachments). The Salesforce REST API can be used to avoid loading an entire attachment into memory. Here is why:

• The REST API returns an attachment BODY as a hyperlink.

• CopyStorm downloads the hyperlink to a temporary file (taking very little memory).

• The temporary file is loaded directly into the target database without ever being loaded into CopyStorm memory. No all database support this feature but those that do minimize CopyStorm memory usage.

There are a handful of tables where CopyStorm uses database views rather than tables. A good example is ActivityHistory (a view combining the Task and Event tables). If Salesforce makes a change which alters the type of length of a column in a table and the column is used in an existing view, PostgreSQL will throw an error like the following when CopyStorm tries to make the change in the database.

• org.postgresql.util.PSQLException: ERROR: cannot alter type of a column used by a view or rule Detail: rule _RETURN on view openactivity depends on column “status”

The process to fix the issue is to:

• Drop the views that depend on the table that needs modification

Yes you can — CopyStorm supports log4j2. This article contains the instructions you will need.

CopyStorm comes packaged with log4j2 — configuring configuring CopyStorm to use it is a two-step process:

• Select the “Use Apache Logging” option on the configuration section of CopyStorm’s Configuration tab.

  • When checked, CopyStorm will use the Apache Commons Logging system rather than the default.

All programming languages, databases, and most applications have limits on the maximum size of values they can store or manipulate. For example, Salesforce has a 2 GB limit the size of an uploaded file, and PostgreSQL has a 1 GB limit on the size of a column value.

When CopyStorm downloads large files from Salesforce, CopyStorm streams the Base64-encoded file body to the file system (which has no size limits other than available disk space). Without using CopyStorm’s Base64 Chunking feature the Base64-encoded file body will be written directly to the database, subject to the size limitations (e.g. 1 GB in PostgreSQL).

The CopyStorm Base64 Chunking feature allows large files to be broken apart into chunks, storing these chunks in a separate table named CopyForceTableFieldChunk. By default, files will be broken into 100 MB chunks – but this is configurable. For example, if the Body field of a ContentVersion record is 500 MB, and the Base64 Chunking feature is enabled, then the 500 MB file will be broken into 5 parts that are stored as 5 separate entries in the CopyForceTableFieldChunk table.

With Base64 Chunking enabled, there are no practical database storage limits to the size of field values in SalesForce objects that can be brought into the CopyStorm database, and CopyStorm is not subject to the maximum size limitations of the database.

The Base64 Chunking feature is disabled by default when CopyStorm is initially configured – to enable this feature, check the

When CopyStorm adds a new column to an existing database table or indexes a new column the operation may have timeout issues if the table has a large number rows. This article describes how to solve this problem by increasing the amount of time CopyStorm will wait for a DDL operation to complete.

The parameter that controls how long CopyStorm will wait for an ALTER TABLE ADD/ALTER COLUMN, CREATE INDEX, or any other DDL type operation is kept in the section of the Configuration tab.

The default DDL Timeout Parameter is 3600 seconds so set the value to a much larger value.

This article provides information on how to capture CopyStorm job execution log data that is output via Standard Error/Out in a text file. This approach is useful for doing advanced troubleshooting in order to capture error symptoms that may not appear within the application GUI or within the logged exceptions of the job history entry found on the “History” tab.

The general approach is as follows:

1. Open a headless CopyStorm script for editing.

   1. This is the script that is scheduled using CRON or Windows Task Scheduler, and is created using the “Batch File” tab of CopyStorm.

2. Create a directory that will be used to save the application logs.

3. Edit the batch script to create a log file with the appropriate name in the established directory and write all job execution output to the file.

Here is an unedited example of a Windows batch script generated by CopyStorm. This script will be modified to output CopyStorm job logs to a file:

The following code is added after the line that begins with “SET CopyStormConfigFile”

• • This code sets a local script variable that establishes the path to the directory where the log files will be stored.

  • The “C:\Capstorm\CopyStormLogs” path should point to your desired CopyStorm log directory path.

Next, the second to last line (starting with “CALL %CopyStormCmd%”) will need to be modified to write to the logfile. The final line will look like this:

This code creates a logfile and redirects stderr and stdout to the file, capturing all job execution logs.

The “CopyStorm_Log_” prefix is used to identify the type of logfile – if you have multiple scheduled jobs, it is best practice to use a different prefix for each scheduled job. For example, “CopyStorm_Hourly_Data_” and “CopyStorm_Daily_Metadata_” if there is an hourly job for data and a separate daily job for metadata.

The example above will create log files with names like:

• CopyStorm_Log_2023-02-21T19-45-32.txt

Here is an edited version of the batch script shown above:

Here is an unedited example of a Linux sh script generated by CopyStorm. This script will be modified to output CopyStorm job logs to a file:

The following code is added after the line that begins with “CopyStormConfigFile=”

• • This code sets a local script variable that establishes the path to the directory where the log files will be stored.

  • The “/opt/Capstorm/CopyStormLogs” path should point to your desired CopyStorm log directory path.

Next, the second to last line (starting with “sh $CopyStormRootDir”) will need to be modified to write to the logfile. The final line will look like this:

This code creates a logfile and redirects stderr and stdout to the file, capturing all job execution logs.

The “CopyStorm_Log_” prefix is used to identify the type of logfile – if you have multiple scheduled jobs, it is best practice to use a different prefix for each scheduled job. For example, “CopyStorm_Hourly_Data_” and “CopyStorm_Daily_Metadata_” if there is an hourly job for data and a separate daily job for metadata.

The example above will create log files with names like:

• CopyStorm_Log_2023-02-21T19:45:32.txt

Here is an edited version of the batch script shown above:

Since CopyStorm keeps a copy of Salesforce metadata in local database tables it is a fairly trivial task to compare metadata from multiple CopyStorm backups.

Problem 1: Compare Table and Field definitions from Two Backups

The key CopyStorm tables required to perform table and field comparisons are CopyForceTable and CopyForceTableField. These are the tables CopyStorm uses to keep the CopyStorm database structure in sync with Salesforce. Rather than bore you with details, download the following SQL script for an example of how to exhaustively search for table and field differences.

• Search for Table and Field Differences Script.

Problem: Compare Salesforce Metadata Objects from Two Backups

This solution compares elements like Apex Classes, Page Layouts, Settings, etc. from two difference CopyStorm backups. The key table in CopyStorm is SFMetaFileProperties since this table contains the most recently backed version of each metadata type. Note that the companion table, SFMetaFilePropertiesHistory, contains historical values for each metadata type. Download the following script for an example of how to exhaustively compare metadata.

There are a number of Salesforce quirks that make creating incremental backup software more difficult than a person might expect. The content of this article will help you understand some of the quirks and give you information that makes you happy that Capstorm backs up Salesforce for you. Note, the text may look familiar to an article published originally on — that’s OK since the blogspot author is one of the founders of Capstorm.

This article is not about how to be a better SOQL programmer. It is about pitfalls in SOQL that are obscure and often difficult to diagnose and understand when encountered.

1. The SOQL WHERE clause cannot be trusted.

2. The SOQL LIMIT parameter is not always optional.

SQL/Server 2016 and later will sometimes complain about adding or altering table columns with a error like:

• SQLServerException: Cannot create a row of size 8063 which is greater than the allowable maximum row size of 8060

If you do not understand what this means then google the message. To fix this issue keep reading.

The simplest fix is to drop the table and let CopyStorm rebuild it automatically during the next backup. SQL/Server will examine the columns in the CREATE TABLE statement and avoid the problem.

The technique for fixing a table without rebuilding it involves converting the longest NVARCHAR(n) columns to NVARCHAR(MAX). This will cause a SQL/Serve table to have a bit more row space for each column that is converted and thus allow the addition of new columns.

If you want to perform the fix by hand, here are the specific steps:

If your CopyStorm database was created before Database Views were introduced then are likely many database tables in your backup that could be represented as database views instead. For safety, CopyStorm will not convert an existing table to a view UNLESS you follow the instructions in this article.

The technique to force CopyStorm to switch to a database view for a table is simple:

• Manually drop the table in the CopyStorm database that should be a view. The next time CopyStorm runs it will create the view.

For example:

• DROP TABLE AccountFeed

How to Login to Salesforce as an External Client App

This page provides instructions for connecting to Salesforce using certificate-based / JWT authentication.

To connect to Salesforce using JWT authentication it is necessary configure an External Client App with a certificate, and to provide the certificate and OAUTH Client Id to CapStorm.

Step 1: Create a Certificate

Create a certificate using CapStorm by selecting the JWT credential type and exporting the certificate file.

To select the JWT credential type:

1. Click on the Edit button for Advanced Parameters.

2. Select Oauth 2.0 JWT in the dropdown.

To Save the certificate:

This will create two files:

1. The certificate and private key are stored in a file CapStorm_JWT.jks in the CapstormConfig folder.

   1. The CapstormConfig folder is a folder next to the application folder.

   2. For example, if CopyStorm is installed in C:\CapStorm\CopyStorm

   3. Then the CapstormConfig folder will be C:\CapStorm\CapstormConfig

Create an External Client App using the PEM file exported in Step 1.

Login to Salesforce and go to Setup. Search for External Client App Manager in the Quick Find box, and click on New External Client App.

Enter app details:

1. External Client App Name

2. API Name

3. Contact Email

4. Callback URL

Then, click on “Create”.

After creating the External Client App, configure app policies to allow access.

Click on “Edit” for App Policies, and update the policy to enable appropriate users. In this example, a Permission Set is used to grant access to the app:

1. Under OAuth Policies, change Permitted Users to:

   1. Admin approved users are pre-authorized.

2. Under Select Profiles and Select Permission Sets:

   1. Select the Permission Set(s) or Profile(s) that should be granted access to the External Client App.

To configure CapStorm, the Oauth Client Id (Consumer key) from the External Client App will be needed.

1. Go to the Settings tab of the External Client App.

2. Under OAuth Settings, click on Consumer Key and Secret.

3. In the Consumer Details page, copy the Consumer Key.

Copy the Consumer Key:

Now, to finish configuring CapStorm:

1. Copy the Consumer Key value into the OAuth Client Id text box.

2. Enter the Salesforce User’s username into the Username text box.

   1. The user must be a member of the Permission Set or Profile configured in the OAuth policy.

3. Enter the Salesforce instance’s host name into the Salesforce Host text box.

To verify the connection, click on the “Test Salesforce Connection” button. If successful, the below window will appear:

The custom certificate checkbox allows users to create and use their own custom java keystore file, separate from the built-in Java JRE keystore. By enabling this checkbox, several new fields will appear in the login form:

The process of configuring OAuth JWT login with a custom certificate is largely similar to the above steps, the main differences being:

• OAuth Public and Private keys will need to be generated manually

• The Certificate will need to be generated manually from the public and private keys

• The certificate will need to be added to a java keystore file or a PKCS12 file

• Keystore info (file path, keystore password, key password, and name) must be entered manually into CopyStorm

After manually generating the above keys and files, the rest of the setup is largely similar to the non-custom certificate flow. Create an external client app, upload the certificate to the external client app, configure the external client app, and finally configure CopyStorm.

For more information on generating keys and certificates for use in Salesforce OAuth JWT, see this .

The Salesforce API presents all SObjects (Account,Contact, etc.) as tables to the outside world even though quite a number of SObjects are really subsets/views of other SObjects. The most common example is Salesforce feed tables — there are dozens of feed tables (AccountFeed, ContactFeed, etc.) that are simply views of the table that contains all feeds: FeedItem.

• The AccountFeed table is really a view on the base table FeedItem.

  • SELECT * FROM FeedItem WHERE parentId IS an Account Record.

There are also cases where the Salesforce API does not support a direct query on an SObject but the SObject can be constructed from other SObjects.

• The ActivityHistory table is a union of the Event and Task SObjects but cannot be queried directly using SOQL.

There are other issues of this kind, but lets get to the important part. When Database Views are enabled in CopyStorm, then CopyStorm will use views in the target database rather than copying a table. This can lead to much faster backups for each table that can be represented as a view.

Database Views are enabled by default in CopyStorm and when enabled:

• If a Salesforce table does not exist in the target database and can be represented as a view, then CopyStorm will create it as a view.

• If a Salesforce table already exists in the target database, then CopyStorm will not re-create it as a view but will continue to treat it as a table.

If Database Views are disabled in CopyStorm:

• Database Views will not be created for new tables.

• If a Database View already exists in the target database, then CopyStorm will not change it.

The toggle that enables/disables Database Views is found in the configuration section of the CopyStorm Configuration tab.

This section is for anyone wanting to know exactly how the logic for determining whether a Database View should be used works.

CopyStorm always stores timestamp value using UTC — exactly like Salesforce. This article demonstrates how to convert a UTC datetime column to a timestamp in your local timezone.

With any backup solution an organization will want to know how their data is secured. Common questions include:

• Where is my data stored?

• How securely is my data transferred over the internet?

• When and where is my data at rest?

For CopyStorm the Salesforce backup security answers are simple and complete:

• Your Salesforce data is stored in a relational database that is created by you and is totally under your control.

  • CopyStorm neither knows nor cares where the data is stored — the database can be in your data center or in a cloud provider like Amazon.

  • The choice and level of security is entirely up to you.

• CopyStorm runs on your computers and transfers all data from Salesforce using HTTPS.

Another common question is:

• What Salesforce and database privileges are required by CopyStorm?

For Salesforce, the only absolutely required privileges are API Access and View Setup and Configuration. The Salesforce API ensures that CopyStorm always respects the privileges of the Salesforce credentials used by CopyStorm to connect. This means that CopyStorm will backup exactly what the privileged credentials would see if they were used with a web browser.

Full details are described in the .

For the target relational database CopyStorm needs to own a schema and does not need visibility outside of its assigned schema. No general DBA privileges are necessary.

While the CopyStorm Batch File can be configured to send email, the best practice is generally to use CopyStorm’s built-in notification settings.

The CopyStorm Batch File tab will generate a script that is useful for running CopyStorm in batch mode. Although the script generator asks for email addresses to notify on CopyStorm success and failure, some installation configurations may cause the generated script to not successfully send a Salesforce backup notification email.

The Batch File tab will generate scripts in a variety of languages. Some of these languages have a built-in way to send email in an operating system independent way, but some do not. When a scripting language (like BAT) does not have a built-in way to send an email then it is the responsibility of the script to implement the behavior.

See the following sections for help with each batch script type.

Hints On Using a Generated BAT File

There is no standard way to send an email from a Windows BAT file, and CopyStorm does not try to guess the technique used by your organization. Generated BAT scripts isolate the “email sending” code to a small subroutine that must be implemented according to your company standard. At the end of a BAT script generated by CopyStorm you will see code that looks like:

:SendEmail REM REM Code your site specific email rule procedure here. REM REM %1 is the email subject REM %2 is the email address that should receive the mail REM %3 is a file containing the body of the email REM

ECHO TODO: Send email to %2 with subject %1 EXIT /B

This code is a subroutine that the script will call each time it attempts to send a Salesforce backup notification email. To finish the script, you need to implement your organization’s email sending logic. BAT file emailing is accomplished using a number of techniques, varying by organization:

• Copying the log file to a directory using a name that indicates when the job completed.

• Using the open source tool to email the status.

The following screenshot indicates where the script changes must be made:

Windows VBS has a built-in way to send an email if the name of an email server is specified. The default email server is GOING TO BE WRONG all of the time, and CopyStorm has no way to guess the correct server. The aspmx.l.google.com server name must be replaced by the name of your company’s email server.

• aspmx.l.google.com is a well-known server that will process email addressed to gmail accounts.

The following screenshot indicates the values that must be changed to match your environment:

The rules for using a BASH script are the same as those for a VBS script — two values must be changed before generating the script:

• EMail Server

• EMail From

See the screenshot under the VBS section above for an example.

The rules for using a Python script are the same as those for a VBS script — two values must be changed before generating the script:

• EMail Server

• EMail From

See the screenshot under the VBS section above for an example.

A common question is “How do I connect to a database running on Amazon (or Azure) from my local laptop?” The answer often is:

• Create an SSH tunnel that will allow you to treat the remote database as if it was local.

Example

We have an instance of MySQL running on an Amazon EC2 instance and want to access it from our local network. Technically what we want to do is:

• Pretend that there is a MySQL instance listening on port 3307 on our local network.

• Have magic forward all communication on localhost port 3307 to a MySQL instance living on Amazon EC2.

The secret sauce requires setting up a forwarding port using ssh. Example:

ssh -L 3307:127.0.0.1:3306 myuser@myhost

This statement will open a terminal prompt on capstormlabs.com and a tunnel listening on 127.0.0.1 that forwards traffic on local port 3307 to port 3306 (e.g. MySQL) on capstormlabs.com. With any CopyStorm product the remote MySQL instance can be reference with a connection string like:

//localhost:3307/myDatabaseName

If you are hungry for more information then see for all the details.

A CopyStorm Snapshot is a database store containing tamper resistant snapshots of records from any Salesforce table where the snapshots are captured based on rules provided by a user. This section explores the architecture and implementation of the CopyStorm Snapshot subsystem and contains recipes for common snapshot tasks.

Good places to start include:

• CopyStorm Snapshot System Overview

• How to Setup the Capstorm Recycle Bin

The minimum Salesforce backup permissions needed for CopyStorm to copy data from Salesforce to a database are:

• API Enabled

• View Setup and Configuration

Almost all of the time, the following permissions should be granted:

• View All Data — this permission bypasses sharing rules and often increases performance.

When CopyStorm runs it tries to find a license on the local machine. If a valid license is not found then it requests one from license.capstorm.com.

If CopyStorm cannot get a license and fails with an error like “Connection Refused” then your firewall is blocking access to license.capstorm.com.

There are two common solutions to this problem:

If your browser is using a proxy to connect to the internet then CopyStorm may have to use the same proxy. See for help setting proxy parameters.

If your company firewall is blocking access to license.capstorm.com, the problem can be fixed by asking your network staff to open port 443 to license.capstorm.com (75.2.67.235).

If the CapstormCustomerKey.txt file is not installed correctly, move the file to a .

In all cases the connection to the Capstorm license server can be validated by following the instructions on .

If your organization will not allow communication with license.capstorm.com, the solution is to send an email to with the following information and we will email you a license file:

On Linux systems the Oracle JDBC driver has a login dependency on the random number generator “/dev/random”. Under certain circumstances, the “/dev/random” generator can take an inordinate amount of time to generate a random number — this can cause unpredictable timeouts when logging into Oracle.

• See for the technical details.

CopyStorm releases after 8.40.1 patch 28-Nov-2017 have fixed this problem internally. If you are on an earlier release, this article describes how to fix the issue.

Open the CopyStorm.sh script in a text editor and add the flag “-Djava.security.egd=file:/dev/./urandom” to the command that launches CopyStorm.

For example, in our tests we added this flag using the following changes to the CopyStorm.sh script:

RAND= if [ -e “/dev/urandom” ]; then RAND=”-Djava.security.egd=file:/dev/./urandom” fi

“$JAVA” -cp “$MYROOT/lib/*” $RAND -Dcopystorm.root=”$MYROOT” -Dhttps.protocols=”$MYPROTOCOLS” $MYAPPLICATION -license $MYLICENSE $*

There are a number of situations where you may want to review Salesforce backup debug logs from CopyStorm — usually this is because something is “wrong” with the Salesforce infrastructure and the problem needs to be narrowed down enough to encourage Salesforce support to rapidly take ownership. Here are a few examples Capstorm has encountered:

• A customer had a table where a query would always time out — even when a single row was selected. The problem was a formula column causing problems with the Salesforce infrastructure. Using CopyStorm’s debug information a sample query exhibiting the behavior (even in the Salesforce Workbench) was quickly found:

  • SELECT id, customFormulaColumnName FROM Contact LIMIT 1

CopyStorm Snapshots are designed so a user can set up new snapshots for a variety of purposes. This tutorial describes a realistic scenario and then walks thru the steps to setup the associated CopyStorm Snapshot Policy.

The legal team for the ACME Beach Club has informed the Salesforce administrator that all Account, Contact, Case, and EMail records related to the account Evil Empire need to be retained and any future changes also need to be retained. To prepare for this situation the Salesforce administrator has already added a formula column in Salesforce, LegalArchivalSet__c, that will evaluate to the string “EvilEmpire” for all of the records the legal team wants to be retained.

The next few sections walk to the steps to setup a CopyStorm snapshot for “Evil Empire” records.

The first step is to create a new snapshot that will only include tables that the legal department indicated.

• Set up the top level parameters for the snapshot including a snapshot policy name and description

CopyStorm supports a number of authentication mechanisms, including:

• Salesforce Username & Password (default)

• The OAuth JWT flow

• SFDX

If Salesforce is using SSO to authenticate users, the Username & Password may not work. If this is happening, there are two options to fix authentication:

CopyStorm includes logic specific to Daylight Savings or Summer Time changeovers.

This logic accommodates both time changes and Salesforce record commit latency issues.

When a CopyStorm job finishes within a few hours of a time change, the next job will re-query for records modified before the time change.

The next job will re-scan an extra 90 minutes of record changes prior to the time change. This ensures that Salesforce commit latency issues and time change issues do not cause missing records in the database.

Jobs that run near a time change will take longer to complete.

This is because the job intentionally reads extra data to ensure that all records are copied. As a result, you will likely see a higher number of “Skipped” records – the job is re-checking data that is already up to date in the CopyStorm database.

Suppose a company has five Salesforce production instances and needs to backup all of them with CopyStorm. How many computers or virtual machines are needed?

This question is a lot like asking “How long is a piece of string?”. Though running many CopyStorm jobs on the same computer will work, the maximum number depends on a lot of factors. Here are a few key questions to ask:

• What is the largest record each job will need to back up?

  • The primary determining factor in the amount of memory required by CopyStorm is the size of the largest record that must be retrieved from Salesforce.

    • For example, if your Salesforce contains a 500MB Attachment then CopyStorm will need around 1300MB of memory at runtime (500MB for the record from Salesforce, 500MB for the prepared database record, and a bit more for overhead).

    • Except for Attachments and ContentVersions, CopyStorm can usually run in 100-120MB (you will see larger usage in most cases because the system allows it and the application uses more memory to avoid frequent garbage collection).

  • You can discover the size of your largest record in the Salesforce GUI and use this information to estimate CopyStorm memory usage.

• How often will each backup run?

  • To backup multiple Salesforce instances, where CopyStorm jobs are scheduled to run at the same or overlapping times the total amount of memory required on your computer may need to be increased.

  • Though running two jobs may not double memory usage, the difference will be material.

• What is the throughput to/from the internet on the computer?

  • The #1 performance bottleneck for CopyStorm is typically the rate at which Salesforce responds to queries.

  • For a single CopyStorm job it is unlikely that throughput to/from the internet will be a problem, but running many CopyStorm jobs simultaneously may swamp a computer’s Ethernet capabilities.

• What is the throughput to/from the database where CopyStorm backups will be stored?

  • The #2 performance bottleneck for CopyStorm is the rate at which data can be written to the target database.

The short answer is that you can keep adding scheduled CopyStorm jobs until you are unhappy with the performance. Running several jobs at the same time will rarely be an issue, but running dozens at the same time may require revisiting your computer’s memory and Ethernet performance.

To create a schema-only CopyStorm backup, you can:

1. Start with an empty database.

2. Go to the “Configuration” tab.

3. Select the “Guardrails” configuration section.

4. Check the “Schema Only Backup” option.

5. Press “Start Copy”.

This will cause CopyStorm to start a new backup, create tables in the database, and copy no table data!

After the CopyStorm backup has completed, the backup database will contain an empty Salesforce schema that is compatible with CopyStorm/Restore.

If new tables or fields are added to the Salesforce instance, running CopyStorm again with the same configuration will update the database with the new or modified tables and fields.

By default, CopyStorm does not store Salesforce or database passwords. To enable this option, select the “Store Passwords” option when saving a configuration file. When checked, CopyStorm will maintain Salesforce password security by encrypting Salesforce and database passwords in the save file.

What if I Want to Store Passwords in a Password Manager?

CopyStorm includes an extensible system which allows the usage of third-party password managers to store Salesforce passwords. The default password storage locations include:

• Directly within the CopyStorm save file (this is the default).

• Within a KeePass file named “passwords.kdbx” located in any of:

  • The user’s home directory.

  • The CopyStorm installation directory.

  • A subdirectory of the CopyStorm installation directory named “config”.

  • Any directory adjacent to the CopyStorm installation directory.

To use your own password manager, please see our tutorial titled .

To enforce Salesforce password security using the KeePass password manager to authenticate with Salesforce, you will need:

• A kdbx (KeePass v2.x) formatted password file containing the username and password of the Salesforce user.

  • For more information on the KeePass password manager, see

  • The user’s Security Token may be either:

    • Appended to the user’s password.

After installing Perl with the kpcli dependencies:

• Put the passwords.kdbx file in one of the locations referenced above.

• Put the kpcli.pl script in one of the locations referenced above (make sure the file is named “kpcli.pl”).

• Make sure the script is installed correctly by running the command:

When running on Windows using Java 1.9 or higher, you may see popups within CopyStorm and CopyStorm/Restore displaying incorrectly. This can occur when using multiple monitors that are configured to use different display scaling settings – a common configuration when running on a laptop computer that is connected to an external display.

The underlying issue that causes this behavior has to do with how Windows displays applications at different scaling settings. To fix the issue, you can:

Step 1: Open the File Properties for Java

CopyStorm may be launched by either javaw.exe or java.exe – if using the Windows version, they will be located at:

• [CopyStorm]/jre/bin/javaw.exe

• [CopyStorm]/jre/bin/java.exe

If there is no jre folder in the CopyStorm installation, then you will need to find the system-installed Java executables.

Right-click on the executables and select ‘Properties’.

1. Select “Compatibility”

2. Select “Change high DPI settings”

1. Check “Override high DPI scaling behavior.”

2. Set “Scaling performed by:” to “System”

After saving / applying the settings (and restarting CopyStorm), the application should display everything correctly!

Salesforce stores the Attachment and ContentVersion bodies as BASE64 strings — a common question is “How can I turn a database Attachment into a file my application can read?”. Since the data is stored as a BASE64 string, this is equivalent to “How do I convert a BASE64 string into its binary equivalent?”.

The answer to this question is database dependent.

Oracle

To perform BASE64 to binary conversion in Oracle:

• Download the base64decode() function from the link on oracle-base.com.

• Use SQL to convert the body of Attachments to their binary format:

  • SELECT id, BASE64DECODE(body) AS rawbody FROM Attachment

You may want to use the BASE64DECODE() function to create a permanent view to the Attachment or ContentVersion table.

To convert a BASE64 column to a text column in SQL/Server:

To convert a BASE64 column to a binary column in SQL/Server:

CopyStorm copies both active and deleted records from Salesforce to the target database by inspecting the Salesforce recycle bin. When a deleted record is discovered its “isDeleted” column is set to true in the target database.

The retention policy for deleted records in the target database is set using the Advanced tab parameter “Delete Older Than”. Available values can range from:

• Remove deleted records immediately.

• Remove deleted records after they have been deleted in Salesforce for 7 days.

• Never remove deleted records.

There are also situations where a record is deleted in Salesforce but is not discovered by CopyStorm. This situation may occur when:

• A user deletes records in Salesforce and empties the recycle bin BEFORE CopyStorm has run.

• A system administrator empties the global recycle bin.

• A Salesforce API program does a hard delete on Salesforce records.

  • A hard delete bypasses the Salesforce recycle bin mechanism.

When a Salesforce record is hard-deleted it is usually wise to use the CopyStorm/Medic Fix IsDeleted tool to mark the appropriate records as deleted in the target database. For information on when and how the CopyStorm/Medic tool should be used see .

While the brave at heart will export data from the CopyStorm Snapshot subsystem using raw database SQL queries, most people will want some help from a GUI. If you happen to be a “brave at heart database person” then jump to the article on CopyStorm Snapshot Data Storage.

For the Rest of Us…

The CopyStorm Snapshot Explorer tool is designed to find and export data from a CopyStorm Snapshot. The tool is launched using a hyperlink next to the snapshot of interest in the CopyStorm GUI.

For this tutorial we will:

• Launch the Snapshot Explorer

• Search for Contacts in a snapshot that match specific criteria.

• Export the found contacts to a CSV file.

See the following articles for additional help.

For developers, a useful way to provide CopyStorm with Salesforce credentials is via a SFDX username or alias. When an SFDX username is provided, CopyStorm calls out to SFDX to create the a Salesforce session token, relying on SFDX to do all of the OAuth dirty work!

Step 1: Change the CopyStorm credential type to SFDX Username or Alias.

Step 2: Enter an SFDX username or alias.

As a final step, optionally test the connection using the Test Salesforce Connection button.

As a best practice, we recommend using an SFDX alias rather than a username in order to support easier reuse when new scratch orgs are created.

Upgrading CopyStorm can be a simple task if you follow one rule:

• Do not store site specific customizations or configuration files in the CopyStorm distribution directory.

If you have set up your Capstorm software by following our Best Practice Guide, you should be ready to continue with the upgrade. If you have included site specific files in the CopyStorm distribution directory then move them into their own directory before starting the upgrade.

To upgrade a CopyStorm installation:

• Download the latest CopyStorm release from https://www.capstorm.com/download

• Rename the existing CopyStorm root directory to a new name.

• Unzip the CopyStorm distribution archive.

• Move the unzipped CopyStorm directory to the same location as your previous CopyStorm installation.

• Copy any site-specific configuration files from the previous installation directory to the newly moved installation directory.

  • Be sure to include the file, if it is installed in the CopyStorm product directory.

You’re finished! CopyStorm distributions have always been backwards compatible with configuration files from previous versions so there is nothing else to change.

WARNING: DO NOT unzip a CopyStorm distribution over an existing installation of CopyStorm.

Metadata tables in a CopyStorm database are primarily copies of the equivalent Salesforce metadata calls plus mappings from Salesforce to CopyStorm names for tables and fields. These tables are the “CopyForce” tables found in the CopyStorm database.

CopyForceTable

CopyForceTable maps the Salesforce API table name to the CopyStorm created database table name. It contains the same fields as a Salesforce DescribeSObjectResult. Where possible CopyStorm uses the same name, but there are several reasons why CopyStorm may use a different table name:

• Names that are the same as reserved words in the target database will be modified.

  • For example, User is renamed to User2 in SQL/Server.

• Table and column name size limitations in Oracle dictate shortening longer names.

For more information, see

CopyForceTableField maps table field names to CopyStorm created field names. It contains the same fields as a Salesforce field.

For more information, see:

CopyForceTableChildRelationship maps parent table names to child table names. It contains the same fields as a Salesforce ChildRelationship.

For more information, see:

CopyStorm does its best to keep column definitions in the target database exactly the same as the corresponding definitions in Salesforce. Normally this happens automatically as CopyStorm performs required ALTER TABLE/COLUMN statements on the target database. This article describes what to do when an ALTER fails. An ALTER may fail for several reasons:

• A column data type in Salesforce changes. Few databases support this conversion automatically.

  • For example, changing a String to a Number.

CopyStorm’s Snapshot System creates tamper resistant snapshots of Salesforce records under conditions that are specified by a customer. Examples:

• Snapshot all deleted records.

• Snapshot modifications to closed Opportunity records.

• Snapshot modifications to Case records where the custom file archiveStatus__c is set to Active or Continuous.

When a new version of CopyStorm is released, adjustments to the internal CopyStorm data model may be required. By default, CopyStorm automatically handles database version upgrades. To disable automatic schema upgrades, refer to the .

CopyStorm’s default behavior is to display a Schema Upgrade Notice dialog to the user:

From this dialog, you can:

• View all pending schema upgrades.

• View specific details about a schema upgrade, including the underling SQL to be executed.

In Salesforce all Date/Time data is stored in UTC — . UTC is not a time zone but rather a time standard used commonly around the world.

For practical purposes, we can assume that UTC is the same as the older GMT (Greenwich Mean Time) standard and is the current time in London, UK.

CopyStorm also stores all Date/Time values in UTC and it is up to an application to convert UTC values to the timezone of each application using the data. This makes a lot of sense for systems with users accessing the same data from wildly different timezones. Each user expects to see Date/Time values displayed in their local time zone!

This section lists a few techniques for converting from UTC to the local timezone of the computer hosting the CopyStorm database.

Example query:

SELECT systemmodstamp ,DATEADD(hh,((DATEDIFF(ss,systemmodstamp,GETDATE())-DATEDIFF(ss,systemmodstamp,GETUTCDATE()))/3600), systemmodstamp) AS LocalTimestamp FROM Account

There are two things to note with this solution:

In general, CopyStorm uses the minimal number of API calls possible to do a backup.

Detailed information can be found in this table:

Salesforce has recently announced their plans to deprecate the SOAP API login method. The purpose of this article is to outline the impact to the CapStorm applications that will result from this change to the Salesforce platform. More information about this change to the Salesforce platform can be found .

When using CopyStorm, the only Salesforce credential type that uses the Soap API login is the "Username and password" option. Note: This does not include OAuth Username and Password.

This change to the Salesforce platform will have a few noteworthy impacts on the CapStorm family of applications. These changes will only affect version 10.65.1 (V5.4 of the CapStorm Container) and newer:

• CapStorm applications using the Username and Password Salesforce login type will fallback to Salesforce API version 64.

The first time a programmer brings a CopyStorm database into ERwin (or another data modeling tool) one of the first questions asked is:

• Where are the relationships between tables?

The “database equivalent” question is:

• Why doesn’t CopyStorm generate FOREIGN KEY constraints?

The short answer to this question is:

The addition of Foreign Key constraints would dramatically increase the complexity and cost of extracting data from Salesforce without improving the performance of database queries.

To connect to Salesforce using the , CopyStorm can be configured with:

1. The OAUTH Consumer Key and Certificate.

2. The username and endpoint for the Salesforce instance.

JWT Authentication can be enabled in Salesforce by following . The overall process is to:

1. Create a Connected Application.

The CopyStorm Snapshot subsystem stores record data in database XML column types and many database professionals may not be familiar with how to write SQL queries to read snapshot data. Though we do not expect many people to write their own SQL queries against CopyStorm snapshots, we have include a query generator to help a programmer to get started.

The basic procedure is:

• Open the Snapshot Explore for the snapshot you want to query.

• Select a table and the columns you want in the query.

For normal tables, SQL/Server has a hard limit of 1024 columns. This article explains what to do if your Salesforce table has more than 1024 columns.

SQL/Server will support up to 30,000 columns per table but to achieve this columns above the 1024 limit must be created as SPARSE columns. A sparse column is a column whose value is stored in a special column that contains an XML document. Naturally, sparse columns are more expensive to search and update than regular columns but they do provide a way to get past the 1024 limit.

The produced to fix a problem table is:

1. Turn a table into a wide table by adding a new column to hold sparse column values.

2. Identify which columns to change to sparse columns.

Having multiple CopyStorm jobs run on the same database at the same time is perfectly OK if the two jobs are processing different Salesforce tables. However, what if two CopyStorm jobs are running at the same time on the same set of tables? How can this happen?

• CopyStorm is scheduled to run every 30 minutes.

• A CopyStorm job starts at 12:00.

• At 12:30 the first CopyStorm job is still running because there were a LOT of updates in Salesforce.

Since, in nearly all cases, Salesforce is a live system that is always changing, it can be difficult to prove that a CopyStorm backup is complete. There are, however, techniques that can be used to gain confidence that a backup is “complete.”

One common approach sites take is to compare record counts reported by Salesforce with record counts in the CopyStorm backup. This approach works pretty well as long as the following facts are understood.

• Salesforce does not include archived or deleted records in reported record counts. CopyStorm backs up both of these record types and may report more records than indicated by Salesforce.

• By default, CopyStorm will not back up a record until it is unchanged for approximately 2 minutes. There is a good reason for this default behavior and it is because of the “” feature. This means that Salesforce may report slightly more records than CopyStorm

To connect to Salesforce using a Time-Base One-Time-Pad (TOTP) Authenticator code, CopyStorm can be configured with:

1. A TOTP Authenticator App Code.

2. The username and password for the Salesforce instance.

TOTP authentication can be enabled in CopyStorm by selecting the “One-Time Password Authenticator” credential type:

When connecting to Snowflake using the “private_key_file” and “private_key_file_pwd” parameters, Copystorm may report an error with the text:

• PBES2 SecretKeyFactory not available

This error indicates that the certificate file is encrypted, and that the Java installation used by CopyStorm does not support the encryption used to secure the certificate file.

This is a known issue in the Snowflake driver (details can be found here) – with multiple possible solutions.

The simplest solutions are to:

• Update the CopyStorm Java version to version 21 or later – Java 21 includes the required libraries.

• Modifying the Private Key File to be unencrypted. While this is not advised due to security concerns, it will resolve the issue by bypassing the problem encryption logic.

A more complex solution is to:

• Update CopyStorm with an encryption library that adds support for the encryption type.

The remainder of this article details the “complex solution” – integrating a third-party library with CopyStorm to enable the missing encryption algorithm. Instructions in this document are for the BouncyCastle security provider.

BouncyCastle is an open-source standard cryptographic library. It can be added to CopyStorm to enable support for additional encryption functions, including the PBES2 function that is missing in Java versions 20 and earlier. For more information about Bouncy Castle, please see .

To add the BouncyCastle PKIX library to CopyStorm, there are three steps:

1. Add the library JAR files to the CopyStorm lib/ directory.

2. Add a JVM argument in the CopyStorm.bat or CopyStorm.sh launch script.

3. Update the Java installation’s java.security file to enable BouncyCastle.

BouncyCastle jars can be downloaded here:

You will need to download two JAR files:

• bcprov-jdk18on-[version].jar

• bcpkix-jdk18on-[version].jar

To add the JARs to CopyStorm, copy them into the [CopyStorm]/lib/ directory.

The following JVM argument is added to the CopyStorm.bat or CopyStorm.sh script:

• -Dnet.snowflake.jdbc.enableBouncycastle=TRUE

To add the parameter in Linux:

• Modify this line of CopyStorm.sh:

  • “$JAVA” -Dcsapp.class=CopyStorm -jar “$MYROOT/lib/CapstormAppLoader.jar” “$@”

• Adding the parameter:

  • “$JAVA” -Dnet.snowflake.jdbc.enableBouncycastle=TRUE -Dcsapp.class=CopyStorm -jar “$MYROOT/lib/CapstormAppLoader.jar” “$@”

To add the parameter in Windows:

• Modify this line of CopyStorm.bat:

  • SET JVM_ARGS=

• Adding the parameter:

  • SET JVM_ARGS=-Dnet.snowflake.jdbc.enableBouncycastle=TRUE

This step requires modifying the java.security file to add BouncyCastle support to Java.

The java.security file is located in the Java installation directory. For example, in the Windows release of CopyStorm, it can be found at:

• [CopyStorm]/jre/conf/security/java.security

To add BouncyCastle:

• Open the java.security file.

• Find the lines that begin with “security.provider.”.

• Add a new line to the end of the list.

• The new line will look like:

Changes will take effect the next time CopyStorm is launched.

• Close any running instances of CopyStorm.

• Start a new CopyStorm instance by running the CopyStorm.bat or CopyStorm.sh script.

If your site is using a self-signed certificate to establish SSL (HTTPS) connections you may see an error like the following when attempting to connect to Salesforce:

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
                at sun.security.validator.PKIXValidator.doBuild(Unknown Source)
                at sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
                at sun.security.validator.Validator.validate(Unknown Source)
                at sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
                at sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
                at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
                ... 26 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
                at sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
                at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
                at java.security.cert.CertPathBuilder.build(Unknown Source)
                ... 32 more

This type of error typically means that your network is using a self-signed certificate. To make CopyStorm (or any other Java based application) work with your certificate, you must add the certificate to Java’s trusted store file “cacerts”.

To automatically add the certificate

• Open the “Help” menu in the top left of the application GUI and select the “Load Network Certificates” option:

• This will open the “Certificate Viewer” popup where you can view, approve, or export the necessary certificates:

• Approve the certificates by clicking the “Approve” button. This will create a keystore file called “CapStormKeyStore.jks” in the OR the current user’s home directory that will contain the approved certificates. CapStorm applications will use this keystore file for SSL traffic.

• Get a copy of your certificate.

  • This will be a CER or CRT file (example: mycompany.crt).

• Copy the certificate to the certificate directory in your CopyStorm installation:

  • [CopyStormRoot]/jre/lib/security

If your environment requires that multiple certificates be installed, you will need to import each certificate using a unique alias:

• ....\bin\keytool -import -keystore cacerts -file firstcertificate.crt -alias firstcertificate

• ....\bin\keytool -import -keystore cacerts -file secondcertificate.crt -alias secondcertificate

If you have encountered this problem, it is better in the long run to use a “cacerts” file for a Java runtime that is installed globally on your computer. The procedure is the same, but the “cacerts” file is in a different location. If you take this approach then remove the “jre” directory from CopyStorm and the application will use the system default Java.

Here is a with more details — please reference it if you need more help.

When metadata backup is enabled for CopyStorm a table named SFMetaFileProperties will be created in the target database to hold metadata backups. In addition a table named SFMetaFilePropertiesHistory will be created to hold past revisions of metadata.

The columns in SFMetaFileProperties mimic the FileProperties data structure returned from the Salesforce Metadata API:

The companion table “SFMetaFilePropertiesHistory” has the same structure as SFMetaFileProperties but with two additional columns:

It is doubtful that you will want to access these tables directly. However, CopyStorm’s companion product, CopyStorm/Restore, uses these tables to provide a metadata report experience that can be performed with surgical accuracy.

This article explains how Capstorm license management works and provides tips on diagnosing license management problems.

When a Capstorm product connects to Salesforce it also checks for a valid product license in the file “license.txt”. The “license.txt” file will be located in the same directory as the script used to launch the Capstorm product. If a license.txt file is not found or a valid non-expired license is not found in the file, the Capstorm product will issue a request to the Capstorm license service to retrieve (or generate) a license.

When issuing the license request to the Capstorm license server, the CapstormCustomerKey.txt file will be used to identify the installation. If the CapstormCustomerKey.txt file is not valid, a message indicating the issue will be shown.

If a Capstorm product fails to retrieve/generate a license it is usually because the connection is blocked by a firewall rule. Diagnosing and confirming this condition can be done using a web browser. Here’s how.

To retrieve or generate a license, a Capstorm product makes a https request in the following form:

Where:

• salesforceOrgId is the full 18 character unique identifier of your Salesforce organization.

• productName is the name of the product for which a license is desired.

• productVersion is the version number of the product.

For example:

If the request is successful the license server will return a JSON string in the following form:

Information from the JSON string will be added to the product’s license.txt file.

If a firewall is blocking access to the Capstorm license server then pasting a license request into a browser will almost always provide a meaningful message. If you receive a “Connection Refused” response then a firewall is likely blocking access to license.capstorm.com. You need to tell your network administrator to open ports 80 and 443 to license.capstorm.com (75.2.67.235).

Here are a few URLs you can click on to test the connection to the license manager:

This article explains how CopyStorm optimally decides which records are copied from Salesforce.

The Most Important Point

CopyStorm’s default behavior is to perform an incremental copy from Salesforce every time it runs. There are no special parameters to set — just use the defaults and CopyStorm will incrementally update your Salesforce backup.

How CopyStorm Selects Records to Copy

CopyStorm minimizes the number of records processed during each run by using record timestamps stored in the CopyStorm database from previous runs.

For example, if the most recent timestamp for an Account in the CopyStorm database is 14-Jul-2013 at 12:00:00.000 then there is no need to read records from Salesforce that were modified before this timestamp.

The following diagram illustrates how CopyStorm reads records for each selected table:

The details are a bit more complex than the above diagram — for example, records are not read one at a time in the code. However, this is the exact logical process followed by CopyStorm.

Advanced Timestamp Details

This section provides a bit more detail on exactly how timestamp values are used by CopyStorm. You do not need to grasp this section to understand how CopyStorm works.

Salesforce has an interesting quirk — records are not committed to a Salesforce database in timestamp order. The implications of this are:

• At 13:01 a SOQL query indicates that there are 1000 records created earlier than 13:00.

• At 13:02 the same SOQL query may indicate that there are 1001 records created earlier than 13:00.

It seems that records written to Salesforce “take a little while” to be committed to the database AND are committed in no particular order.

CopyStorm overcomes this Salesforce quirk using two parameters on the Advanced tab:

• The SFDC Commit Latency parameter indicates how long to wait for Salesforce to commit records. The default is 300 seconds.

  • This means that, by default, CopyStorm will not read a record from Salesforce until it has a timestamp at least 300 seconds old.

• The Timestamp Latency parameter indicates how many seconds to subtract from the most recent timestamp found in CopyStorm.

To sum up the details, the SOQL used by CopyStorm really looks like:

• SELECT * FROM TT WHERE TT.systemmodstamp>=(MostRecent – “TimestampLatency”) AND TT.systemdstamp <=(CURRENT_TIME – “SFDC Commit Latency”)

In Salesforce a query of the form “SELECT * FROM TT” will often fail with a timeout error on large tables. To avoid this problem CopyStorm runs every SOQL query with a LIMIT. For most tables the LIMIT value is set to 20,000 — but is configured to be smaller for a handful of tables known to have problems with this high of a limit.

The default LIMIT value can be overridden on the Advanced tab using the “Row LIMIT per Query” option — however, changing this parameter generally results in non-optimal behavior.

In the case where all records in a batch have exactly the same timestamp, CopyStorm dynamically increases the LIMIT value until the end of records is reached or not all records have the same timestamp. When this type of adjustment is required it is rarely noticed unless a Salesforce table has 100,000’s of records with exactly the same timestamp.

If you would like to know about further Salesforce quirks CopyStorm has to manage (and be convinced that you don’t want to write your own CopyStorm) take a look at this blog post: .

CopyStorm supports reading Database credentials from a user-supplied Database password manager.

This feature was introduced to eliminate the need to:

• Change the saved password in many CopyStorm configuration files.

• Provide the password to CopyStorm on the command line due to security rules.

In addition to security benefits, this feature allows you to use your own company-wide password manager.

The process to add a new Database password manager to CopyStorm is to:

CopyStorm can be configured to provide a headless Salesforce backup by running in batch/background mode and leveraging the built-in scheduler on your computer. This article will show you how.

The base script used to launch the CopyStorm UI can also be used to start CopyStorm as a headless Salesforce backup application (required for a batch job). To configure the CopyStorm command line:

1. Save your CopyStorm configuration file using the “Encrypted” and “Save Passwords” options on the CopyStorm “Save As” dialog.

   • This will create a file named “MySettings.copyStorm” (using the name you choose for the file).

During the initial backup of Salesforce instances which include large numbers of old Tasks or Events, CopyStorm may experience repeated Salesforce timeout issues when running SOQL queries on these tables. This article explains approaches for solving these problems.

Once the initial backup of the Task and/or Event tables is complete, the Salesforce timeout issue goes away and the recommendations on this page are no longer needed.

Note that these recommendations apply to any Salesforce table, but this issue is rarely seen elsewhere.

Within the Salesforce infrastructure there are multiple levels of data storage — within this section we will refer to them as Tier-1, Tier-2, and Tier-3. Frequently used data (and most tables) tend to be kept in fast Tier-1 storage. Data in some tables (like Task) are migrated to slower storage as they age and are not queried by a user. Eventually most old Task records will be migrated into the slowest storage tier, Tier-3, and become fairly time consuming to query. When data has been migrated to Tier-3 storage, the only way to cause it to move back go faster storage is to attempt to read the Tier-3 data. When old data has been referenced enough times it will be “promoted” into faster storage — even old Task records can migrate from slow Tier-3 storage back to Tier-1 storage if they are referenced frequently.

When CopyStorm first attempts to backup the Task table it runs a query like:

SELECT * FROM Task ORDER BY SystemModStamp ASC LIMIT 20000

If you have a security gateway that manipulates XML returned from Salesforce before passing it along to CopyStorm, you may see a Salesforce SOAP Entity Not Recognized error like:

• ConnectionException: Found invalid XML. could not resolve entity named ‘deg’ (position: START_TAG seen …1133;敌湈त 2;喚솕驘&# 20077;°… @122:9536)

This error means that your gateway has converted a standard XML character representation (e.g. “°”) into its equivalent HTML entity (in this case “°”). XML parsers are not required to understand the hundreds of entity tags used by HTML (and many, including the one used by the core WSC Salesforce libraries, do not).

Due to this issue, starting with version 8.37.1, CopyStorm has augmented the standard WSC parser to understand nearly all HTML entity tags. If you encounter an entity tag missed by CopyStorm, this article explains how to fix the issue.

To add a new entity to the core Salesforce XML parser:

This articles explores situations where CopyStorm appears to miss an update or insert of a record from Salesforce and the solutions to correct the CopyStorm database. The root causes tend to be one of the following:

• Salesforce permission changes altered what columns are visible to the credentials used when running CopyStorm.

• Salesforce permission changes altered what records are visible to the credentials used when running CopyStorm.

• There is a problem altering the CopyStorm schema to match Salesforce.

• The “Salesforce believes in Time Travel” problem.

When CopyStorm discovers a new column from the Salesforce metadata it automatically creates the new column in the database and assumes that it must be a new column in Salesforce. Here is the scenario where changing Salesforce permissions make it look like CopyStorm missed updating column.

• The Salesforce credentials used by CopyStorm do not have permission to see the column Contact.MyCustom__c.

• On Monday the Contact table is backed up with CopyStorm.

• On Tuesday morning an administrator changes permissions to allow the CopyStorm Salesforce account to see Contact.MyCustom__c

• On Tuesday afternoon an incremental backup of the Contact table happens.

How do you fix this issue? There are two easy approaches:

• Use CopyStorm/Medic’s tool.

• Run CopyStorm with the following options:

  • Enter the name of the problem table in the Tables to Copy box.

  • Enable the Force Updates option.

Since CopyStorm can only backup Salesforce records it can see, if a Salesforce permission change makes new records available they normally will not be picked up until their timestamp changes. Here is how to make CopyStorm rescan all records in a table and pick up missing records.

• Enter the name of the problem table in the Tables to Copy box.

• Enter 1/1/1990 in the Modified Since option.

• Run a Copy

• CopyStorm will rescan all records and insert any missing records into the database.

If a field is altered or added in Salesforce and CopyStorm finds it impossible to make the same change to the database then CopyStorm will record an error and refuse to backup data to the table until the error is resolved.

The quickest way to fix this problem is:

• Enter the name of the problem table in the Tables to Copy box.

• Click on the Force Rebuild option.

• Run a backup and CopyStorm will drop the problem table, recreate it, and then repopulate it with data from Salesforce.

If the quick fix is too drastic for your case, the see the article.

First, this problem does not occur unless CopyStorm internal parameters are modified but it is included in this article for completeness (and because it is rare condition that few people know about). The end of this section describes how to make it possible for the “Time Travel” issue to occur but you may have to wait months for it to happen!

CopyStorm achieves fast incremental backup of Salesforce by using the timestamp on Salesforce records. Logically CopyStorm does the following each time it runs:

• Read the most recent timestamp from the CopyStorm database and call it “T”

• Run SOQL that looks like: SELECT * FROM TableName WHERE timestamp >= T

The “Time Travel” problem is that Salesforce DOES NOT always commit records in timestamp order. This means that a record whose timestamp is 12:02 may show up in Salesforce BEFORE a record with a timestamp of 12:01. This does happen often but when it does it is likely that any program doing incremental copies based on a timestamp may:

• Save all records modified at timestamp 12:02 or earlier in the first run MISSING the one record modified at 12:01.

• Start searching for new records using timestamp 12:02 in the second run and NEVER SEE the 12:01 record.

Fortunately it is possible to pretend that Salesforce always writes records in timestamp order by not processing records that have been modified in the past 2 minutes. How did we pick 2 minutes? We worked with a lot of high volume sites and also simulated massive Salesforce loads that reproduced the problem.

Occasionally a person evaluating CopyStorm will decide that it is skipping a record when performing an experiment like:

• Backup all Contacts

• In Salesforce, modify a Contact.

• Immediately run CopyStorm. It will copy no records because the change is less than 2 minutes old.

When the evaluator emails support@capstorm.com we point them to the original time travel post and suggest that they run CopyStorm again. Of course, it picks up the changed record the second time.

If you are doing CopyStorm work on a sandbox and are willing to live with the “time travel” problem you can tell CopyStorm to pretend that a safe amount of time to wait is 1 second. Here is how:

• Select the section of the CopyStorm Configuration Tab.

• Click on the Edit... button next to Internal Parameters.

• Set SFDC Commit Latency and Timestamp Latency to 1.

The CopyStorm “Copying…” tab has real-time metrics which help identify a components that are limiting the speed of a database backup. The metrics, located in the lower right hand corner of a “Copying…” tab are described in this article.

There are two primary factors which limit the speed of a CopyStorm job:

• The rate at which Salesforce delivers data to CopyStorm.

• The rate at which the target database accepts data from CopyStorm.

Metrics highlighted in the following image provide data indicating which component is the bottleneck at your site:

Metrics are divided into three sections:

In a balanced system:

• The “S” threads will be blinking blue almost all of the time.

  • This means that a steady stream of data is being delivered from Salesforce.

• The “D” threads will be grey most of the time but blinking blue rapidly.

• The data throughput values will be less than 50ms and the database write speed will be less than 20ms.

In a system where the database is the bottleneck:

• The “S” threads will often turn grey.

• The “D” threads will be blue most of the time.

• The database write speed will be large.

  • This value is elapsed time for a single request to the database to write a block of records.

In a system where Salesforce is the bottleneck:

• The “S” threads will be solid blue most of the time.

  • This means that they are waiting for a response from Salesforce.

• The “D” threads will be grey most of the time.

• The first data throughput number will be large.

The most common issue we see is that CopyStorm and the database are not in the same data center. The underlying issue is that the protocols used to talk to relational databases require close proximity between the application and the database for great performance. Until we discover a way to increase the speed of light, CopyStorm running in India and talking to a database in Canada will give slow performance. You can solve this problem by running CopyStorm and its corresponding database in the same data center.

CopyStorm will write data to a database as quickly as Salesforce delivers it. If the “S” threads are grey, the “D” threads are blue, the data throughput numbers are large, and you are sure that CopyStorm and the database are in the same data center; then the database is likely overloaded. Using system monitoring tools on the database computer will quickly pinpoint the issue — often the database computer will be page-faulting like crazy.

If the CopyStorm “S” threads are solid blue, the “D” threads are almost always grey, and the first data throughput number is large; then Salesforce is not delivering data rapidly. Even with a modest internet bandwidth of 20M/s, CopyStorm can easily transfer 20,000 – 30,000 accounts per minute. If you are sure that your available bandwidth is high then see your system administrators. There may be network software inspecting and slowing packets from Salesforce.

Though CopyStorm’s runtime requirements are modest, it cannot control demands made by other applications. Use a local system monitor to see which applications are using a high amount of CPU time. On a balanced system CopyStorm will always be consuming CPU — reading from Salesforce and writing to the target database.

To ensure seamless integration between CapStorm and Salesforce, you’ll need to configure a Salesforce Connected App.

This setup enables CapStorm to access Salesforce APIs and retrieve necessary data.

This guide walks you through the steps to configure a Connected App

and use it to see exact API responses from Salesforce, particularly for support purposes.

1. Log in to Salesforce with an administrator account.

2. Navigate to the Setup Menu: