Prior to running CopyStorm for the first time, create a fresh SQL/Server database.

SQL/Server Connection String

The only difficult part when copying Salesforce to SQL/Server is constructing the connection string. Clicking on the Connection String Builder will open a form that, if you are running SQL/Server using a standard configuration, will construct a connection string for you.

The connection string can be verified by pressing the “Test SQLServer Connection” button.

The general form of a SQL/Server connection string is:

• [server-name[\instance-name][:port-number]][;property=value]*

If you would like to see more information regarding SQL/Server connection strings or are not familiar with Microsoft’s type 4 JDBC driver, these resources may help:

Capstorm includes a MySQL driver from the to communicate with both Oracle’s MySQL product and with MariaDB.

Since MariaDB is provided by the original authors of MySQL and strives to maintain compatibility with Oracle’s MySQL, the driver packaged with CopyStorm is probably the only driver you will need. However, if you are encountering compatibility issues or other problems with MariaDB’s MySQL driver, you can download and install Oracle’s MySQL driver by hand.

CopyStorm does not package this driver with the product due to Oracle’s licensing policy for ISV’s, which makes it difficult to package the Oracle MySQL JDBC driver with a product.

To install the Oracle MySQL JDBC driver:

• Delete the file lib/mariadb-java-client-*.jar from the CopyStorm installation directory.

Initial Step

Prior to running CopyStorm for the first time, create a fresh Oracle database.

Oracle Connection String

The only complicated configuration required in the CopyStorm Main tab is the Oracle connection string. Clicking on the Connection String Builder button will open a form that will construct the connection string for you, given database information.

You will likely need one of these forms:

• myDomain.com:1521:orcl

• myDomain.com:1521:xe

• localhost:1521

The connection string can be verified by pressing the “Test Oracle Connection” button.

The following is a screen shot of a working Oracle setup on one of our test computers:

Initial Steps

Prior to running CopyStorm for the first time, create a fresh PostgreSQL database.

Connection String

The only complicated configuration required in the CopyStorm Main tab is the PostgreSQL connection string. Clicking on the Connection String Builder button will open a form that will construct the connection string for you, given database information.

The general form of a PostgreSQL connection string is:

• [//host[:port]][/database-name]

Example:

• //localhost/CopystormBackupDatabase

By default, a PostgreSQL connection will use the public schema. This can be overridden by entering a schema name.

The following is a screen shot of a working PostgreSQL setup on one of our test computers:

Initial Steps

Before running CopyStorm for the first time, create a dedicated Snowflake database for your Salesforce data.

Connection String

Compared to the other Database types, Snowflake’s connection string is more complex and carries more information. For this reason, it is recommended to use the Connection String Builder to create a valid Connection String.

Connection String Builder

After clicking on the “Connection String Builder” button, a dialog menu will open, as seen below.

The Configuration Tab contains options to modify the CopyStorm runtime, including the list of Salesforce Objects/Tables to copy.

The default view of the Configuration Tab contains core configuration options that are commonly modified.

To view additional options, change the “Configuration Level” dropdown:

• Standard

  • Core CopyStorm configuration and performance tuning.

  • Options used to ensure that CopyStorm does not interfere with other operations (e.g. limiting Salesforce API Call consumption).

• Advanced

  • CopyStorm troubleshooting and logging options.

• Developer

  • CopyStorm application tuning options, advanced network configurations, and query optimization options.

The Guardrails Tab is used to limit the potential impact of individual CopyStorm runtimes and to ensure that CopyStorm jobs do not interfere with other operations.

The Maintenance Configuration Editor is used when troubleshooting replication issues.

The Database Reset Configuration Editor is used to delete and rebuild portions of the CopyStorm database.

Options within the Database Reset Configuration Editor are only used in specific circumstances and should not be left enabled as part of normal backups.

The External Configuration Editor is used to provide external folders used by CopyStorm to read configurations and to store temporary files.

The All Changes Filter Rul e always indicates that a record should be snapshot for a change to any field. Basically, if a caller asks if a record should be snapshot then the All Changes Filter Rule will always respond with “Yes.”

This rule is useful for any table where all changes should be included in the snapshot.

Since the All Changes Filter Rule always says “Yes, this record should be snapshot”, naturally its only configuration parameter is which tables to include in the snapshot.

The Deleted Records Filter Rule indicates that a CopyStorm record should be snapshot only when its IsDeleted flag is set to true. The primary use of the filter is to manage the records added to a recycle bin.

Though CopyStorm is initialized with a policy, Default Recycle Bin, that uses this filter rule, a user can added additional snapshot policies based on a Deleted Records Filter Rule if a non-standard recycle bin is desired.

The only configuration for a Deleted Records Filter Rule is which tables to include in the snapshot.

Welcome to the CopyStorm/View Learning Center.

In this area you will find articles explaining how to quickly create a database combining one or more tables from multiple CopyStorm backups.

Overview

CopyStorm/View builds SQL which combines tables from multiple CopyStorm backups into a new database built from database views.

For example:

• Suppose an organization has three divisions: Americas, Europe, and South America. Each division has its own Salesforce instance, using CopyStorm for backups.

• The CEO is asking for daily reports on opportunity generation and wants to see it across all divisions.

CopyStorm/View helps solve this problem by creating a single database including data from all three divisions by creating views for selected tables that look like:

CREATE VIEW Opportunity( CopStormDatabase, Id, Name, …) AS SELECT ‘Americas’, Id, Name, … FROM AmericasBackup UNION ALL SELECT ‘Europe’, Id, Name, … FROM EuropeAmericasBackup UNION ALL SELECT ‘SouthAmerica’, Id, Name, … FROM SouthAmericaAmericasBackup

This makes finding the answer for the CEO as simple as:

SELECT CopyStormDatabase AS Division, Name, Amount WHERE CreatedDate >= TODAY

If your site does not permit communication with the default Capstorm license server located at https://license.capstorm.com, the configuration file “Capstorm.xml” allows overriding the endpoint.

An example that changes CopyStorm to use a http (non-SSL) endpoint for license management is:

<Capstorm>
    <licenseServer url="https://license.capstorm.com/LicenseStore/GetLicense" />
</Capstorm>

There are a number of other elements that can appear in a Capstorm.xml file, but you should not change without being instructed to by Capstorm. However, if you want to confuse your colleagues you can change CopyStorm’s background color:

<Capstorm>
    <backgroundColor red="255" green="100" blue="15" />
</Capstorm>

Internally, CopyStorm replicates data to Snowflake by creating a CSV file of the records being backed up and uploading the CSV file to Snowflake. It may be required to change this staging location to meet business requirements. CopyStorm allows customers to specify a different Snowflake stage by following this process:

Create a xml file named “SnowflakeDatabaseBuilder.xml”

Then place the contents below into the xml file.

<SnowflakeDatabaseBuilder>
    <SnowflakeStage value="@clientProvidedStagePath"/>
</SnowflakeDatabaseBuilder>

You will need to replace “clientProvidedStagePath” with the Snowflake stage’s path, then place this file into CopyStorm/config directory and relaunch CopyStorm.

Note: This feature is available in CopyStorm versions 10.56.5 and above.

The CopyStorm/Medic Delete Salesforce Records tool has the ability to run batch mode without the need of opening the application GUI and click through the steps of the tool. Execution of the tool in this manner requires a CopyStorm/Medic configuration file that already has all desired configuration options and filters selected and saved. For more information regarding the setup of this tool, see here.

Example Usage

This example is for Windows installations of CopyStorm/Medic. For Linux/Mac installations, change CopyStormMedic.bat to CopyStormMedic.sh

CopyStormMedic.bat -tool deleteTool -run "/path/to/config.copyStormMedic"

While commonly used schema options within CopyStorm are directly exposed in the CopyStorm GUI, a number of schema initialization parameters can only be provided by providing CopyStorm with a JDBCDatabaseBuilderConfig.xml file.

To use the options detailed in this page, create a JDBCDatabaseBuilderConfig.xml file and copy it to one of the CopyStorm Configuration File Directories.

Capitalize Column & Table Names

The following XML will cause CopyStorm to create a Salesforce data table and column names in ALL CAPS.

<JDBCDatabaseBuilder>
<!-- 
    Set these values to "true" to capitalize table and column names.
-->
	<CapitalizeTableNames enabled="true" />
	<CapitalizeColumnNames enabled="true" />

</JDBCDatabaseBuilder>

Remove Table & Column Postfixes

The following XML will cause CopyStorm to remove Salesforce-standard postfixes from database table and column names.

Oracle 12c added support for extending the allowed length of VARCHAR2 columns from 4000 to 32,767 bytes but made the discovery of whether this feature is enabled require privileges that a normal Oracle user is not granted. This article explains how to tell CopyStorm that the Oracle feature “max_string_size=EXTENDED” is enabled without requiring special Oracle permissions.

For your reference, if an Oracle user has enough privileges then a simple query can be used to determine if the EXTENDED Varchar2 feature is enabled.

• SELECT name,value FROM V$PARAMETER WHERE name=’max_string_size’

It is likely that your account will return a “Table or View not Found” when running this query because of your limited permissions.

To inform CopyStorm that the EXTENDED feature is enabled simply create a file named Oracle.xml and drop into the your private CopyStorm configuration directory specified on the Advanced tab or into the directory [CopyStormRootDir]/config. The content of the Oracle.xml file should be:

<Oracle>

<DatabaseProperty name="max_string_size" value="EXTENDED" />

</Oracle>

Restart CopyStorm and future VARCHAR2 columns will be up to 32,764 bytes in length. You may wonder why the limit is 32,764 rather than 32,767. The reason is that Salesforce defaults to a UTF-8 Extended character set and each character can take up to 4 bytes.

Though every Snapshot Policy has a default record retention rule, it is often convenient to have a custom retention rule for specific tables. Example:

• By default, retain records in a snapshot for 90 days.

• Retain Account, Contact, and Opportunity records for 365 days.

• Retain Case records for 2 years.

• Do not retain EmailMessage records.

The following two screens illustrate how to set up this policy.

Clicking on the Advanced button launches an editor to override the default record retention rule. The following example sets up the scenario described at the top of this page.

A SQL Predicate Filter Rul e is useful when other filtering techniques are insufficient and it is possible to write a database SQL fragment to identify records that match a snapshot policy. It is rare to need this filter rule because filtering can typically be accomplished using the graphical Column Expression Filter Rule.

A SQL Predicate Filter Rule has a single configuration parameter: a fragment of database specific SQL will be added to the WHERE clause of a SQL statement generated by CopyStorm. Example:

• SELECT Id FROM Account WHERE Id IN (…) AND (configuration parameter from SQL Predicate Filter)

If the argument to a SQL Predicate Filter Rule was “RecordAge__c>46 OR alwaysSnapshot__c=1” then SQL statements generated by CopyStorm would look like:

• SELECT Id FROM Account WHERE Id IN (…) AND (RecordAge__c>46 OR alwaysSnapshot__c=1)

A SQL Predicate Filter Rule also includes a rule for which tables to include in the snapshot.

The Run Snapshot Policy Retention Rules feature is used to force a snapshot to look for records that should be deleted from the Snapshot based on its retention rules and then delete the records. This happens automatically for active Snapshot Policies as part of the regular CopyStorm backup process, so this feature is most useful for inactive Snapshot Policies.

The Run Snapshot Policy Retention Rules feature is accessed from the right-click context menu on the CopyStorm Snapshot tab.

There are only two parameter options when running retention rules.

• Search Batch Size — number of records that will be searched at once for potential deletion.

• Deletion Batch Size — number of records that will be deleted in a single database transaction.

Like most database systems, the CopyStorm Snapshot system will operate most efficiently when its has statistics about what is stored in snapshots. Example:

• If CopyStorm knows what tables have been snapshot by a policy then it can limit searches to just these tables.

• If CopyStorm knows how may records are in a snapshot it can generated more efficient SQL queries.

For the most part, CopyStorm Snapshot updates statistics without any user intervention. This page, however, describes how to force CopyStorm Snapshot to update statistics if the most accurate statistics possible are desired or if the statistics have been corrupted. Note: it is unlikely that you will ever need this feature.

The Update Statistic s feature is accessed from the right-click context menu on the CopyStorm Snapshot tab.

In the Refresh Statistics dialog there is the option to refresh statistics for a single snapshot or for all snapshots in the system. The following image illustrates what the screen may look like after refreshing statistics for a single snapshot.

The Database Networking Configuration Editor contains networking and SQL parameters used when communicating with the CopyStorm database.

The CopyStorm Snapshot tab is the launch point for creating, editing, using, and exploring CopyStorm snapshot data sources. A CopyStorm Snapshot is a database store containing tamper-proof snapshots of records from any Salesforce table where the snap-shots are taken based upon rules specified by a user.

CopyStorm Snapshots can be used to store many types of data.

• A Snapshot can be used to create a unlimited duration recycle bin or a recycle bin where data retention policies vary by table.

• A Snapshot can be used to permanently save a copy of Salesforce records on demand.

• A Snapshot can be used to track all versions of critical Salesforce records.

Each CopyStorm Snapshot has policies which determine which table and which records should be snapshot. Like the core CopyStorm backup process, Snapshot policies can be applied automatically as a regular part of a backup or can be applied on demand.

The CopyStorm Snapshot tab lists all currently defined snapshots, can be used to create a new snapshot, and is the launch point for most Snapshot operations including:

To get started with CopyStorm Snapshots, please review the following:

Once you have the basics down try one of these tutorials:

Finally, jump to any of the following reference articles:

The Database Indexing Configuration Editor is used to enable or disable automatic database index creation.

A Snapshot Policy Filter Rule contains the logic which determines if a database record should be processed by the snapshot policy. There are several types of policy filters built into CopyStorm Snapshot.

• All Changes — this filter always indicates that a record should be processed. Use it when all changes to records should be snapshot.

• Column Expression — with this filter a rich expression based on columns in a record can be used to determine if a record should be snapshot.

• Deleted Records — this filter only matches records that have been deleted in Salesforce. This filter is used to build recycle bin type policies.

• — this advanced filter supports injecting user specific SQL fragments into a query that determines if a record should be snapshot.

The Filter Rule for a Snapshot Policy is specified on the Main Tab of the snapshot editor.

Parameters specific to the selected Filter Rule Type can be set on the Filter Rule Parameters tab.

Nearly all Filter Rule Types support the concept of a White and/or Black list for selecting the tables to be processed by the policy.

• A White List indicates tables to include in the snapshot.

• A Black List indicates tables to exclude from the snapshot. All tables not on the black list will be included.

Typically the technique used for selecting tables for a snapshot is either by White List or Black List. However, if you are brave (and using regular expressions) both a White and Black list may be useful.

The following example (from the All Changes filter rule type) illustrate the components of the White/Black list editor.

• The pull-down menu indicates whether tables should be selected by White list, Black list, or both.

The dialog for selecting tables is nearly the same for a White or Black list. The following example is for a White list.

A Column Expression Filter Rule is a rich graphical tool for specifying the conditions under which a record should be added to a CopyStorm snapshot. The interface is patterned after the Salesforce Report Writer expression builder and should feel familiar to most users.

The first step, after selecting the Column Expression option from the Filter Rule Type pull-down, is to launch the column expression filter editor.

Clicking on the Edit Column Expression Filter button will launch the editor.

The following example creates a filter that finds records where:

• the Name field contains “Louis”

The screen image shows the dialog after the Refresh Preview button has been clicked. Notice that the searched database has multiple accounts with the similar names that matched the filter.

After saving changes and returning to the policy editor, a summary of the filter will be shown. To change the filter simply click on the Edit Column Expression Rule again.

The list of available columns is determined as the intersection of all columns from table names matched by the policy’s Include Tables list. If you see a single column named Id then the policy is including all tables in Salesforce and the only common column name is Id. To increase the number of available columns return to the policy editor page and adjust the list of included tables.

This issue only occurs when there are no matching records in the CopyStorm database in any of the selected tables. The solution is to modify the query.

The Snapshot Explorer is a tool for viewing and exporting data from a CopyStorm Snapshot. The tool is launched from the CopyStorm Snapshot tab.

The Snapshot Explorer has four stages, indicated by the navigation bar at the top of the dialog.

• Start — Select the snapshot to explore.

• Select Table — Select a table from the Snapshot to explore.

• Preview — Build a query and preview the selected data from the Snapshot.

• Export — Export the data from the snapshot in a variety of formats.

In the Select Table stage, select the table you wish to explore and optionally select the columns in the table to view. Note that you can always return to this stage in order to update the selected columns.

In the Preview Stage, optionally create a filter that limits the records to view or export. The filter is created by dragging and dropping elements from the left hand side of the screen to the big white box on the right.

In the example screen shot a filter for finding records with a capstorm email domain has been specified and then the Refresh Preview button was clicked.

In the Export Stage:

• Review and update the list of columns to export.

• Click on the Start Snapshot Export button and the system will display a typical Save As dialog.

By default, only the most recent version of a record will be written in the export. To override this behavior and write ALL VERSIONS of a record to the export check the box to Include All Snapshot Record Versions of a Salesforce Record. Of course, even if this option is checked, a record still has to pass the filter criteria from the Preview stage.

The Save As dialog which appears after clicking on Start Snapshot Export will look like your system’s default dialog with the addition of a file format selection section.

When the result of an export is loaded into your favorite spreadsheet it will look like:

The Run Snapshot Policy feature is used to run a snapshot policy against existing data in a CopyStorm snapshot on demand. This process is useful in a number of circumstances.

• Suppose a new snapshot for all Cases older than 2 years was just defined. Before activating the snapshot policy (so it will run automatically when a backup happens) you may want to snapshot all existing cases in the CopyStorm database that are older than 2 years.

• Your compliance department has told you to snapshot all records related to ACME, Inc. because they have filed a breach lawsuit against the company. In this case the snapshot policy that you define may run just once — on demand — and never be scheduled to run automatically.

The Run Snapshot Policy feature is accessed from the right-click context menu on the CopyStorm Snapshot tab.

There are just a few options to run a Snapshot Policy by hand.

• Scan Records From sets a start date from which records will be searched. By default the start date will be the date of the earliest record in the corresponding CopyStorm table.

• Scan Records Thru sets the last date on which records will be searched. By default the end data will be the most recent date in the CopyStorm database.

• # Concurrent Searches limits the number of tables that will be processed concurrently. You can set it to a large value (like 200) but be aware that a large number will consume considerable database resources.

While a Snapshot Policy is running the screen will continuously update with progress information. At any point in time you can cancel the snapshot process and only records snapshot so far will be preserved in the database.

The following screenshot illustrates how the dialog will appear after a snapshot policy has run to completion. Note that are several reasons that a record may be processed but not snapshot.

• If a record fails to match the filter associated with the snapshot policy then it will be skipped.

• If a record matches the filter associated with the snapshot policy but is already in the snapshot with exactly the same data then it will be skipped.

The Batch File tab is used to generate a script that can be used to launch CopyStorm as a batch job. Generated scripts encapsulate everything needed to run a CopyStorm job, including:

• The location of CopyStorm.

• The location of the configuration file defining backup parameters.

For many sites a generated batch file is not necessary due to built-in command line options in the main CopyStorm launch script allowing batch mode execution. Example command lines include:

• sh CopyStorm.sh -run /path/to/my/configFile.copyStorm

• CopyStorm.bat -run C:\path\to\my\configFile.copyStorm

The core function of the Batch File tab is to generate a script file that will:

• Run CopyStorm using a specific configuration file from any directory using just the name of the script file.

• Capture any exception information from CopyStorm to “do something” with the data. By default the script emails exception information to a user but this can easily be adapted to other purposes.

If you have configured the , the process for generating a script file is simple. This is the approach used nearly 100% of the time:

• Use the “Script Format” menu to specify the desired scripting language — bash, bat, python, or vbs.

• Click on the “Generate Script” button.

• Click on the “Save Script” button and save the script to a file.

If your site cannot use the built-in Notification system the process for generating a script file is:

• Use the “Script Format” menu to select one of the script formats marked as Advanced.

• Enter an email address for the notification sent each time the script runs.

• Enter an email address for the notification sent if CopyStorm reports an exception.

• Enter the name of your local SMTP server.

After these steps are completed, you own the generated script file and can change it in any way you prefer.

If you have trouble getting the email portion to work properly, please read .

The Core Configuration Editor is used to specify which objects should be replicated to the database, and to control how long records will be retained in the database after being deleted in Salesforce.

Select Objects

The “Select Objects” dropdown selects the objects which will be copied from Salesforce to the database.

The easiest way to configure which objects to copy is to select one of the pre-configured options:

• All Objects

  • Copy all Salesforce objects that the CopyStorm Salesforce User has access to.

  • Automatically copy newly created Salesforce Objects when they become visible to the CopyStorm Salesforce User.

• All Objects Except for Files

  • Copy all Salesforce objects that the CopyStorm Salesforce User has access to, except for Base64/File objects (e.g. Attachment, ContentVersion).

  • Automatically copy newly created Salesforce Objects when they become visible to the CopyStorm Salesforce User.

• All Objects That Can Be Restored

  • Copy all Salesforce objects that the CopyStorm Salesforce User has access to, except for objects that are known to not be recoverable (e.g. External objects, History objects).

  • Automatically copy newly created Salesforce Objects when they become visible to the CopyStorm Salesforce User.

To configure CopyStorm to copy specific objects, select from the “Manual Selection” options. Selecting a “Manual Selection” will show an “Include” or “Exclude” list. To configure the list, either:

• Type or copy-paste the API names of selected objects into the “Select Objects…” section as a comma separated list.

• Click on the “Select Objects…” button.

  • This will cause CopyStorm to connect to Salesforce to determine the Objects available.

  • A dialog will open allowing selection of available Salesforce Objects.

Manual selection options include:

• Manual Selection – Include List: Choose Objects to Include

  • All selected objects will be copied.

• Manual Selection – Exclude List: Choose Objects to Exclude

  • All objects visible to the CopyStorm Salesforce User will be copied, except for selected objects.

The “Delete Older Than” input controls when to remove records that have been deleted from Salesforce from the target database.

• 0 (zero): This is the default value. Remove deleted records immediately from the target database.

• Blank (no value specified): Mark deleted records as IsDeleted, and never remove deleted records.

• A non-zero number: Mark deleted records as IsDeleted, and remove records when they have been deleted from Salesforce for the number of days specified by “Delete Older Than”.

CopyStorm discovers deleted records by reading the Salesforce recycle bin. If the recycle bin is emptied before CopyStorm discovers the deletion, or if records have been removed with a hard delete, CopyStorm will not know that the records are deleted. This method is used by some applications. If this is occurring, please use Capstorm’s CopyStorm/Medic tool to discover the deleted records and mark the records as IsDeleted in the target database.

If the “Include Metadata” checkbox is checked, CopyStorm will include selected metadata components in the backup. By default, if checked, all metadata components are included.

Clicking the "Edit" button next to this checkbox opens the metadata selection dialog.

The Metadata Selection Mode dropdown offers three options:

• All Metadata Types — All available Salesforce metadata types are backed up. This is the default. Any new metadata types added by Salesforce in the future will automatically be included.

• Include Selected Types — Only the specific metadata types selected from the list will be backed up. Use this to back up a small, known set of types and ignore everything else.

• Exclude Selected Types — All available metadata types are backed up except the ones selected from the list. Use this for broad coverage but need to omit a few specific types. Like "All Metadata Types," any new metadata types added by Salesforce in the future will automatically be included. The exclusion list is applied against whatever is currently available, not a fixed snapshot.

The dialog also includes:

• Include Managed Packages — When checked, metadata from managed packages is included in the backup.

• Delete Metadata History after N days — Controls how long historical versions are retained. Leave blank to keep history indefinitely; set to 0 to keep only the current version with no history.

The Salesforce API Configuration Editor is used to choose which Salesforce API will be used to copy data.

Data API Options

The Data API Options section is used to configure the primary Salesforce API that will be used to copy data.

• Salesforce Threads

  • The number of Salesforce Objects that will be backed up at the same time by CopyStorm.

  • The default number is 3, allowable values are 1 – 20. For maximum performance, the optimal value for this parameter is typically 10.

• Salesforce API

  • The default API to use when querying Salesforce. The default API is REST.

  • Note that CopyStorm will, by default, use REST of any table containing a BASE64 column because this minimizes memory usage for most databases. Common BASE64 tables include Attachment and ContentVersion.

The Bulk API parameters section controls when CopyStorm will use the Salesforce Bulk API rather than the default API.

Note that:

• A Salesforce organization has a limited number of Bulk API calls per 24 hour rolling period. Be aware of your limit.

• The SOAP or REST API will almost always perform better than the Bulk API if performance is measured in the amount of time a user has to wait for a backup to complete.

  • We suspect that this is because Bulk API queries run on a reduced priority thread within the Salesforce infrastructure.

The Metadata Data API Options section is used to configure the primary Salesforce API that will be used to copy Metadata.

• Metadata Threads

  • The number of Salesforce Metadata Types that will be backed up at the same time by CopyStorm.

  • The default number is 5, allowable values are 1 – 10. For maximum performance, the optimal value for this parameter is typically 5.

To configure a S3 partition in CopyStorm/Medic, provide connection information for the target S3 bucket.

CopyStorm will use the AWS Default Credential Provider Chain to connect to S3. Environment variables, Java Properties, IAM Roles, etc… can be used to authorize CopyStorm for S3 access.

To initialize the partition, CopyStorm/Medic will:

• Copy field data for the selected fields into the target bucket, using the folder name as a prefix.

• DROP the database storage used by the selected fields.

When CopyStorm replicates new or updated records, field values for the selected fields will be kept in files in the target bucket.

Record data is kept in a nested structure generated using Salesforce Ids.

CopyStorm supports saving a backup configuration to a file in two formats:

• Plain-text XML.

• Encrypted text (256-bit DES encryption).

Both formats support the optional saving of Salesforce and database credentials. A CopyStorm backup configuration can be completely restored by going to File/Open in the application menu. To save a configuration, go to File/Save in the application menu:

The plain-text XML version of the file is fairly complex and not intended to be edited by hand, but it is possible. However, Capstorm may not provide support for problems arising from a hand-edited configuration file.

To configure a Folder partition in CopyStorm/Medic, provide the target folder:

To initialize the target directory, CopyStorm/Medic will:

• Copy field data for the selected fields into the target folder.

• DROP the database storage used by the selected fields.

When CopyStorm replicates new or updated records, field values for the selected fields will be kept in files in the target directory.

Record data is kept in nested folders generated using Salesforce Ids.

The Database Query Optimization Parameter Editor controls optimizations used when generating SQL queries for the CopyStorm database.

The Salesforce Network Configuration Editor contains networking and SQL parameters used when communicating with Salesforce.

The SOQL Query Optimization Parameter Editor controls optimizations used when generating SOQL queries for Salesforce.

The Main Tab is used to manage connection information and credentials for Salesforce and the target database.

Specifies the type of target database. Available database targets depend on the version of CopyStorm you have installed.

These parameters are used when authenticating with Salesforce using a Username and Password.

• For certificate based authentication, see the .

• For SSO-based authentication, see .

The Field History Configuration editor is used to enable and configure CopyStorm field history tracking.

CopyStorm field history tracking supports the tracking of nearly all field changes in every Salesforce object.

If the “Track Field History” checkbox is selected, the default behavior is to track all changes to all fields. To limit field history tracking or to customize retention periods for field history changes, click on the “Edit” button for advanced Field History configuration.

Field history changes are kept in a database table named “CopyForceFieldHistory”. For details see:

Field History is controlled by three major elements:

CopyStorm includes a number of useful tools that can be found under the “Tools” menu in the top-left corner of the application GUI.

The following tools are included:

• SOQL Query Tool (Salesforce)

• SQL Query Tool (Database)

• Skipped Record Log Explorer

The CopyStorm/Medic Refresh Formulas tool replaces the values of formula columns in CopyStorm with their current value in Salesforce.

This tool is typically used when the definition of a formula is modified in Salesforce.

CopyStorm minimizes the amount of records copied from Salesforce by only copying records whose modification date has changed. When a formula column is modified in Salesforce record modification times do not change so CopyStorm does not pick up the new formula values for older records. There are two ways to update this data in the CopyStorm database:

• Performing a forced update on the CopyStorm database using CopyStorm. This technique works but will cause every column in the CopyStorm tables to be read from Salesforce and written to CopyStorm.

• Using this tool to limit reads and updates from Salesforce to formula columns. This technique can be many times faster than a forced update.

The CopyStorm/Medic Fix IsDeleted tool identifies records that have been deleted in Salesforce, removed from the Salesforce recycle bin, but not marked as deleted in CopyStorm.

This situation occurs when:

• A user deletes records in Salesforce and empties the recycle bin BEFORE CopyStorm has run.

• A system administrator empties the global recycle bin.

• A Salesforce API program does a hard delete on Salesforce records. A hard delete bypasses the Salesforce recycle bin mechanism.

The CopyStorm/Medic Refresh Columns tool replaces the value of one or more columns in CopyStorm with their current values in Salesforce.

This tool is typically used when the definition of a formula is modified in Salesforce or when a previously hidden column becomes visible to the user CopyStorm uses to authenticate with Salesforce.

When one or more columns in CopyStorm do not match the current value in Salesforce there are two methods which can fix the problem:

• Perform a forced update on the CopyStorm database using CopyStorm. This technique works but will cause every column in the CopyStorm tables to be read from Salesforce and written to CopyStorm, including columns whose data is already synchronized.

• Use this tool to limit reads/updates from Salesforce to specific columns. This technique can be many times faster than a forced update, and does not re-synchronize data that is already synchronized.

When the Salesforce REST API returns a record with a BASE64 column, the BASE64 data is returned as a hyperlink that can be used to read the raw bytes. Experience has shown that it is possible for the hyperlink returned by Salesforce to return a HTTP 404 (not found) or an HTTP 500 (internal lightning platform error) rather than the expected HTTP 200 (OK). When this happens:

• The BASE64 data cannot be read by the REST API.

• The same operation using the SOAP API results in an UNKNOWN_EXCEPTION or an unparsable SOAP body

The Base64 Sanity Checker tool tests the hyperlinks returned by the REST API and reports the ID of records where the hyperlink fails (e.g. Does not return a HTTP 200). It is often wise to check each reported record using the Salesforce GUI to make sure that the data is really available. If you find a corrupted Attachment record report it to Salesforce.

The CopyStorm/Medic Table Editor tool is for non-DBA people who want to remove data or tables from a CopyStorm database. Why would someone want to do this?

• You may want to drop a specific table so CopyStorm will rebuild it the next time it runs. This case comes up when a column definition in Salesforce has changed and the corresponding change is not supported by the target database.

• You may want to drop a Salesforce Feed table, like AccountFeed, so CopyStorm will replace it with a database view the next time it runs.

• You may want to make CopyStorm completely forget about a database table it created. This action drops a table and all references to it in the CopyStorm metadata.

The Fix DocLinks tool repairs records in the Salesforce ContentDocumentLink table. The Salesforce ContentDocumentLink table has query restrictions that make it impossible to efficiently perform an 100% accurate incremental backup. There are two primary reasons that block an 100% accurate backup.

1. The ContentDocumentLink table cannot be queried only on its SystemModStamp field.

2. ContentDocumentLink permission values can be changed and there is no way to find the changes efficiently because of reason #1.

In practice most customers will either never run this tool or run it on a weekly basis.

To access the Fix DocLinks GUI:

When partitioning tables containing large volumes of data or Base64-encoded content, the partition creation process can fail mid-way due to various issues, such as:

• Application freezes or crashes

• Transient network errors

• Resource exhaustion (e.g. memory or CPU spikes)

The CopyStorm/Medic Find Dirt tool finds data in Salesforce fields that should not be possible. Capstorm provides this tool because, as a Salesforce instance ages, it will often accumulate data that does not match the field definition. Here are a few examples:

• A string field is limited to 80 characters but has a few records where 96 characters are stored.

• A numeric field is limited to zero digits after the decimal point but Apex code inserted records where digits appear.

This tool will report the table, record id, column name, and problem description for any field that does not match the Salesforce field definition.

To access the Find Dirt GUI:

Though MySQL claims support for all UTF-8 characters, experience indicates that not all MySQL variants support the full extended 4 byte UTF-8 character set. When MySQL fails to recognize an extended UTF-8 character and error like this will be displayed:

• java.sql.BatchUpdateException: Incorrect string value: ‘\xF0\x9F\x98\x8A’ for column ‘CommentBody’ at row 48

To get around this issue, you can configure CopyStorm to replace unsupported 4 byte UTF-8 characters with a substitution.

Note: The extended 4 byte UTF-8 character set contains like 100 variations of a smiley face, trees, hamburgers, etc…

Character mappings are defined in a “UTF8Tool.xml” configuration file that can be placed in either the “config” subdirectory of the CopyStorm installation or in a custom config directory you have created. The following UTF8Tool.xml will cause CopyStorm to replace the 4 byte UTF-8 characters for two smiley faces to be converted to their more old fashioned counterparts:

On rare occasions a site may receive an UNKNOWN_EXCEPTION from CopyStorm. In CopyStorm’s log, the error will look like:

10-Feb-17 09:02:11: [UnexpectedErrorFault [ApiFault exceptionCode=’UNKNOWN_EXCEPTION’ exceptionMessage=’An unexpected error occurred. Please include this ErrorId if you contact support: 303701882-51722 (1187285824)’ extendedErrorDetails='{[0]}’

This error is generated by Salesforce’s server when it encounters a problem that Salesforce code does not know how to handle. Most of the time the issue will resolve itself after a couple days as Salesforce fixes their own internal problems. If the problem does not clear up automatically then this article may help you resolve the issue.

In Capstorm’s experience, to get Salesforce support to pay attention to UNKNOWN_EXCEPTION issues a problem has to be reproduced in an application supplied totally by Salesforce. When this error is being given by Salesforce we need to:

• Isolate the SOQL query in CopyStorm that is causing the problem.

If CopyStorm is reporting an OPERATION_TOO_LARGE error, it will be included in the CopyStorm log. The error will look like this:

09:02:11: [UnexpectedErrorFault [ApiFault exceptionCode=’OPERATION_TOO_LARGE’ exceptionMessage=’exceeded 20000 distinct ids’ extendedErrorDetails='{[0]}’ ] ] Before diving into the root cause, this is the most common solution:

• Grant the ViewAllData privilege to the Salesforce credentials being used with CopyStorm.

The root cause of this exception can be found in the Salesforce description of the error:

• OPERATION_TOO_LARGE

The CopyStorm/Medic Scan Timestamps tool performs an exhaustive comparison of Salesforce versus CopyStorm record modifications timestamps. This tool is provided for two reasons:

• It can be used to provide assurance that CopyStorm is copying all records properly.

• If Salesforce has an “after update” trigger on a table where the trigger updates the source table, this tool can be used to help track down the problem trigger.

Comparing modification timestamps from Salesforce with CopyStorm is a trickier task than one might think — here are a few of the reasons why:

There are a number of XML configuration files that can be used to override default CopyStorm behavior. This article describes the approach CopyStorm uses to find configuration files.

When CopyStorm searches for a configuration file, it looks in four places:

1. Configuration files embedded within the CopyStorm application.

   1. CopyStorm always looks here first.

The CopyStorm/Medic Log Manager tool supports the viewing, management, and exporting of the results of previously run CopyStorm jobs.

Starting with version 7.*.*, CopyStorm stores the status of each job into tables in the CopyStorm target database. The CopyStorm/Medic Log Manager is a convenient tool for viewing, managing, and exporting the contents of these tables. In addition to Log Manager, regular RDBMS tools can be used to access CopyStorm’s logs.

Full documentation for the CopyStorm/Medic Log Manager tool can be found .

The database tables used for RDBMS access to CopyStorm and CopyStorm/Medic logs can be found .

Though it is possible to delete a Snapshot Policy, usually a site will simply want to make the policy inactive. When a Snapshot Policy is deleted both the policy definition and all data snapshot by the policy are deleted from the CopyStorm database.

The Snapshot policy deletion tool is launched using right-click context menu on the CopyStorm Snapshot tab.

The CopyStorm/Medic Permission Checker will find Salesforce tables and fields that are visible using the Salesforce metadata API but are not visible for backup using the current Salesforce credentials.

It is a common misconception that a member of the Salesforce Administrator group can see all table and fields in a Salesforce instance. In practice, there are even many standard Salesforce fields that a Salesforce Administrator cannot view by default. The Permission Checker helps you find these tables and fields.

Running the Permission Checker involves clicking on a single button – Start Scanning Metadata.

• The Salesforce user credentials provided to the Permissions Checker must have access to read the Salesforce metadata definitions for each object.

• Use of the metadata API requires the system permission “Modify Metadata Through Metadata API Functions.”

The CopyStorm/Medic Global Parameters page sets parameters that apply to all CopyStorm/Medic jobs.

The Global Parameters page can be found by selecting the “Other Tools” dropdown in CopyStorm/Medic, and clicking “Global Parameters” in the dropdown list.

In certain configurations it may be necessary to run specific SQL statements or set database specific options each time a connection to a database is established.

The configuration file “Oracle.xml” contains parameters that control connections made to Oracle. The following configuration directives are supported:

• InitSQL — specify a line of SQL that will be run each time a new connection to Oracle is established. There is no limit to the number of InitSQL elements.

The following Oracle.xml file will cause CopyStorm to run the command:

• alter session set ‘_optimizer_rownum_pred_based_fkr’=FALSE

This page provides details into the command line options for running snapshot policies for CopyStorm in headless mode. This feature was introduced as part of the 10.65.1 GA release of CopyStorm. For more information regarding running snapshot policies, see here.

Overview

The command line functionality mirrors the same options and functionality as seen in the CopyStorm application using the "Run Snapshot Rules Now" option including:

• Start and end date ranges for records to snapshot

• Number of concurrent searches (number of objects snapshot in parallel)

• Search batch size

• Snapshot per batch

This example is for Windows installations of CopyStorm. For Linux/Mac installations, change CopyStorm.bat to CopyStorm.sh

CopyStorm supports the tracking of nearly all field changes by checking the “Track Field History” box on the Advanced tab. When field history is enabled, default behavior is to track updates to EVERY field in EVERY table except for a handful of Salesforce auditing fields.

If the default behavior captures more data than your organization prefers, this article explains how to limit what is tracked.

Background

Fine control over field tracking is managed by a configuration file named FieldHistoryRecordFactory.xml. The file can be placed in either the “config” subdirectory of the CopyStorm installation or in the directory indicated in the CopyStorm configuration file. The built-in configuration file has rules excluding field tracking for a number of Salesforce audit fields and all formula columns.

It looks like:

<FieldHistoryRecordFactory>

    <IncludeFormulas status="false" />

    <ExcludeField field="SystemModStamp" />
    <ExcludeField field="LastModifiedDate" />
    <ExcludeField field="LastModifiedById" />
    <ExcludeField field="CreatedDate" />
    <ExcludeField field="CreatedById" />
    <ExcludeField field="LoginTime" />
    <ExcludeField field="LastViewedDate" />
    <ExcludeField field="LastReferencedDate" />

</FieldHistoryRecordFactory>

If you create your own configuration file, model it after the built-in file without repeating any of the built-in rules.

Like CopyStorm table selection, there are two basic approaches to controlling which fields are tracked:

• Telling CopyStorm which fields should be tracked.

• Telling CopyStorm which fields should NOT be tracked.

If CopyStorm is told which fields to track the rules indicating which fields should NOT be tracked are ignored and only fields CopyStorm is explicitly told to track will be included.

To only track field history on specific tables, include the IncludeTable configuration directive.

The following FieldHistoryRecordFactory.xml will limit field history to the Account, Contact, and MyCustomObject__c tables:

To track fields in all tables EXCEPT for a few then use the ExcludeTable configuration directive.

The following FieldHistoryRecordFactory.xml will exclude ApexPages, ApexTriggers, and MyCustomObject__c from field history tracking:

The following FieldHistoryRecordFactory.xml will cause CopyStorm to only track the “Name” field on “Account” and the “FirstName”/”LastName” fields on “Contact”:

The following FieldHistoryRecordFactory.xml will cause CopyStorm to not track the “WebSite” field on “Account”, and to not track the “Age”/”Birthday” fields on “Contact”:

Default CopyStorm behavior is to not track changes to formula column values. The following FieldHistoryRecordFactory.xml will enable formula value history tracking:

The CopyStorm/Medic Count Records tool performs a comparison of object record counts in Salesforce versus CopyStorm. This tool is provided as a sanity check for backup validation, to provide assurance that CopyStorm is copying all records properly.

Run Count Records in a GUI

To access the Count Records GUI:

• Select “Other Tools” from the top-level menu.

• Select “Database Validation”, then “Count Records”

The UI for the Count Records tool contains four parts:

• The left-hand side of the dialog is used to select the tables to scan.

• The top part of the dialog has the following parameters:

  • of Tables to Process Concurrently.

    • Specifies how many tables the tool will process in parallel. Higher values may improve performance but increase resource usage.

  • Output To CSV (checkbox).
• The lower part of the dialog displays the progress of a record count analysis as it happens.

• If an error is detected, a button will appear in the lower right hand corner. Click on the button to see error details and/or export the errors to a file.

• Windows

  • CopyStormMedic.bat -tool countRecords -run configFile.copyStormMedic

• Linux / Mac OS

When replicating Salesforce data and schema to the CopyStorm database, CopyStorm will create database tables and columns which mirror the object and field structure in Salesforce. For example, there is a Salesforce object called “Account”, with a String field called “Name”. This means that the CopyStorm database will have a table called “Account” with a VARCHAR or CLOB field called “Name”.

CopyStorm partitions use a technique called Vertical Partitioning to change the CopyStorm database schema by separating the database table for an object into multiple tables. Of course – CopyStorm, CopyStorm/Restore, and CopyStorm/Medic are fully compatible with CopyStorm partitions.

For example, by default, the CopyStorm database may have an “Account” table with these columns:

1. Id

2. SystemModstamp

3. Name

4. BillingStreet

5. BillingCity

6. BillingState

If the “Billing” fields are moved to a new partition, the database will have multiple tables:

1. The “Account” table, with these columns:

   1. Id

   2. SystemModstamp

   3. Name

Virtually any field can be moved to a partition if needed – with the exception of a small number of fields that require special handling.

• The “Id” database column is always present on every partition.

• A handful of fields are required to remain on the primary partition:

  • IsDeleted

  • Salesforce Modification Timestamps:

Partitions may be created for virtually any object – the only exceptions are:

• BigObjects

• Tables that are included in the .

Background

Note: This feature is available in Capstorm application versions 10.56.6 and above.

Capstorm applications allows you to add supplementary Java options. These options serve to dynamically alter the operational parameters of the application prior to its initiation. One frequent utilization, which serves as an example throughout this article, is the facilitation of user-driven control over memory allocation for any of Capstorm’s applications. This capability gives users the control to finely calibrate the memory resources dedicated to the operation of Capstorm’s applications.

Configuration File Creation and Placement

Note: Application must be restarted for these configuration files to be applied.

Step 1 – File Creation and initial template setup:

1. First create a XML file named “AppLoader.xml”

Step 2 – File Placement:

Two recommended placement options:

• Option 1: If you followed CopyStorm’s Best Practice Directory Structure then we recommend placing this configuration file in CapstormConfig directory.

• Option 2: Inside of one of Capstorm’s application installation directory there should be a directory called “config”. You can also place this configuration file in the “config” directory.

Note: For versions older than 10.56.6 please visit: on how to configure memory settings.

• Once you have completed “Configuration File Creation and Placement” steps you can simply copy the snippet below into the AppLoader.xml configuration file.

• The example below shows how to set the min and max memory allocations to 300 and 1000 megabytes, respectively.

The AppLoader also allows for custom Java Properties. Below is an example of setting the HTTPS Protocols to TLSv1.1 and TLSv1.2.

The Logging Configuration Editor is used to configure CopyStorm logs that are written to the database, STDOUT / STDERR, or using the Apache Commons Logging framework.

The Database Schema Configuration Editor contains parameters used to configure CopyStorm database schema generation.

CopyStorm Snapshots support an unlimited number of Snapshot Policies with each Policy defined by the Snapshot Policy Editor. The following image lays out the primary sections of the Snapshot Policy Editor.

There are several tabs on the Snapshot Policy Editor.

• The Main tab contains properties common to all snapshots and is where the Filter Rule Type for the snapshot is indicated.

• The Filter Rule Parameters tab contains filter specific data. For example:

The Notifications Tab contains parameters used to control who receives a status email when a CopyStorm job runs and under what circumstances email is sent. CopyStorm supports email notifications via SMTP or via a Salesforce SOAP API call.

When CopyStorm writes data to an S3 partition, each Salesforce record is stored as one or more files in your S3 bucket. The location of every file is derived entirely from the Salesforce record ID. There are no date folders, job ID folders, or table name folders.

The folder hierarchy is constructed by slicing the first 13 characters of the Salesforce record ID into five segments:

As of CopyStorm/Medic 10.54.2, the Base64 Sanity Checker function has been merged into CopyStorm/Medic.

For the latest documentation, please see:

When the Salesforce REST API returns a record with a BASE64 column, the BASE64 data is returned as a hyperlink that can be used to read the raw bytes. Experience has shown that it is possible for the hyperlink returned by Salesforce to return a HTTP 404 (not found) or an HTTP 500 (internal lightning platform error) rather than the expected HTTP 200 (OK). When this happens:

The CopyStorm/Medic Lineage Maintenance Tool is used to view lineage data & invalidate problematic lineage records.

Over time, a Record’s lineage may be invalidated:

• One of the legacy lineage entries may no longer exist in the CopyStorm Database or Snapshot.

• Records may have multiple children, causing split lineage.

For more details on Record Lineage, .

The CapStorm Permission Set Creator tool allows you to configure a permission set that has read access (and optionally write access), create or update the Permission Set in Salesforce, and automatically assign it to the CopyStorm Salesforce user.

The Salesforce user credentials supplied to CopyStorm must have the following permissions in order to use the Permissions Set Creator:

• Modify All Data

• Modify Metadata Through Metadata API Functions