# CopyStorm CopyStorm replicates a Salesforce instance, copying the structure and data to a relational database — SQL/Server, MySQL, PostgreSQL, Oracle, or H2. Once the initial data load is complete, CopyStorm can be run as often as every 5 to 15 minutes to incrementally backup changes. CopyStorm can be downloaded from [www.capstorm.com](https://www.capstorm.com/). # Salesforce Backup Validation The purpose of this guide is to supply best practices to validate a CapStorm backup of Salesforce. This process involves of four steps to be performed in the order listed below: 1. [Permissions verification](#permissions-verification) – Verify that the Salesforce user credentials that are used for backups have access to read all desired Salesforce objects and fields. Objects include Salesforce standard and custom objects, including managed packages. If the analysis uncovers that permissions updates are needed, these permissions gaps are rectified, then [backup updates](#backupupdates) are made to copy all data that was previously inaccessible. 2. [CopyStorm review](#copystorm-configuration-review) – Verify that the application is set up to meet the business’ backup goals. The recommended settings are highlighted in this section, however, each organization may choose to customize the configuration(s). If the analysis uncovers that configuration updates are needed, these changes may need to be tested in a Sandbox environment prior to making Production updates. 3. [Backup validation](#backup-validation) – Verify that all desired records are in the backup. If the analysis uncovers that there are records not present in the backup, these gaps are addressed prior to performing the final backup validation. 4. [Restore testing](#restore-testing) {optional} – Verify the process to recover data from the backup. This optional step follows the steps outlined in a sample runbook, and this often serves as a template for a customized recovery plan. ## Permissions Verification The Salesforce user credentials provided to CopyStorm determine what objects and fields are available for backup. The goal of this step is determine if there are any gaps in the current backup caused by inadequate Salesforce permissions, and, if there are gaps, to update (1) the Salesforce permissions and (2) the backup data set to ensure that all records are copied. There are two primary categories of Salesforce permissions: System Level Permissions and Object/Field Permissions. System Level Permissions impact one or more objects, and these are granted to the Salesforce user directly on the profile or via permissions set. Object / Field permissions impact individual objects and these are granted via permissions set for most standard or custom objects. If the objects are a part of a managed package or are a part of Salesforce licensed features, Salesforce feature of managed package licenses may also need to be assigned to the user in addition to granting object level access. ### System Permissions The following minimum permissions are recommended: * API Enabled * View Setup & Configuration * View All Data * Read Encrypted Data * Query All Files * Knowledge User * Marketing User * Content User * Manage Unlisted Groups * Moderate Chatter Click on the “Main” tab in CopyStorm and click “Test Salesforce Connection.” The window should show these boxes checked: ![](/files/QGoF5GIG22wEGkGea45m) ### Object & Field Permissions Salesforce object and field permissions can be validated by using the CS:Medic [Permissions Checker](/reference/copystorm-medic/permission-checker). This tool is simple to run with a single button click, and the output is a list of objects and fields that the Salesforce user can not currently access. In this example, the Salesforce user can not access data in 12 standard fields and 2 custom fields: ![](/files/KsBFmWsAJnj5e8nMGzbp) If permissions gaps are found, access to the identified objects and fields can be granted directly in Salesforce. Alternatively, CopyStorm’s [Permissions Set Creator tool](/reference/permission-set-creator) will create a “Read All” permissions set to grant access to all objects and fields. Managed packages and some Salesforce features will require additional permissions in addition to the access granted via the CopyStorm created permissions set. Once permissions are updated, run the Permissions Checker again to verify that there are no objects or fields inaccessible to the Salesforce user. If Salesforce field level permissions are updated, a backup update is necessary in order to update older records that were backed up before the field was able to be read by the Salesforce user. This is necessary because the backup process captures records incrementally, meaning that the backup will not automatically re-read records that already exist in the backup database when Salesforce permissions are updated. For example, the Salesforce user in the objects and fields permissions example above was not able to read the data in 2 custom fields. All of the records are already in the backup database, however, the data in these 2 fields is missing. The next step is to update the backup in order to capture the data in these two fields. There are two approaches to choose from in order to update the backup. 1. A full rescan – This option is used if there are many fields that the backup user was unable to see prior to updating permissions or when working with a smaller number of records. 2. A targeted rescan – This option is used if there are few fields that the backup user was unable to see prior to updating permissions or when working with a very large number of records that would make a full rescan time prohibitive. ### Full Rescan The full rescan will re-read all records from Salesforce in order to address field gaps caused by insufficient permissions. To perform this task, use CopyStorm to run a one-time backup using the parameters shown below in combination with targeted objects or all objects. ![](/files/AfUIryHV5HvK3wFcYZR2) A Modified Since value older than the first record created ensures that CopyStorm reads all records from the Salesforce org. This setting ignores the most recent timestamp values in the existing CopyStorm database, instead querying all records. ![](/files/puw4sYn9OManxxblxWfA) Ensure that only “Force Updates” is checked under “Database Optimization.” This ensures that all records will be updated to include field data that was not visible historically due to the Salesforce user’s permissions. Save this Configuration and run. Add to the Notifications distribution list if desired. ### Targeted Rescan The targeted rescan uses CS:Medic to pull only the specific fields that need to be updated. Prior to using this option, have CopyStorm update the schema of the backup database using the “Schema Only Backup” option: ![](/files/mTZl4NLFYjb8yEhlVfJQ) This step ensures that the structure of the object is updated to include the field(s) that are now visible to the Salesforce user. Once the schema has been updated, use the [Refresh Columns tool in CS:Medic.](/reference/copystorm-medic/refresh-columns) ## CopyStorm configuration review This step ensure that CopyStorm is configured for a complete backup including all objects, retention of deleted records, an established rollback period for data changes, and notifications in case a backup encounters an error. Key parameters are outlined below along with CapStorm’s recommended settings. | Setting | Recommendation | Screenshot | | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Objects included |

Create two configurations:

  1. All Salesforce objects except Files & Attachments –
    Schedule this job to run every 4 hours
    (or desired frequency) using CS:Director.
  2. Files, Attachments, and Metadata –
    schedule this job to run once a day
    (or desired frequency) using CS:Director.

All additional settings below should be set
for both configuration files.

|

Configuration 1:

Configuration 2:

| | Deleted Record Retention |

Leave “Delete Older Than” area blank.
This will save deleted records indefinitely.

| ![](/files/81jKR3pRFXzYc7DQrHOe) | | Point-in-Time Record Snapshots |

Activate feature by ticking “Active” then “Save Changes.”

This will automatically version data changes to support
the rollback of a record to a prior state.

Reference Snapshot for more details.

| ![](/files/VgICDXcUINZKDzR0BCQm) | | Notifications |

Specify who should be notified if there are any exceptions.

Separate email addresses with a semicolon.

Include
if desired.

| ![](/files/hQmVV7sIMLiPcipbM4vH) | ## Backup validation This step validate that the backup contains all desired records by using two CS:Medic tools: CountRecords and Scan Timestamps. * CountRecords performs a simple Count query to Salesforce and to the Database and outlines any differences. * [Scan Timestamps](/reference/copystorm-medic/scan-timestamps) performs a deeper analysis, and is typically used to determine the reason for differences identified using the CountRecords tool. Pay close to attention to any discrepancies with a difference greater than 1%. This can be indicative of a Salesforce permissions issue, for example, the User not having access to an Opportunity record type. ![](/files/zFrpeHm2vgrYbWcZobAY) If the CountRecords tab is not open click “Other Tools” in the top menu bar to open the tool. Click “Invert” to include all objects then “Start Scan.” For assistance with this analysis contact . ## Restore testing Annual recovery tests are recommended for Salesforce disaster recovery preparation. An alternative is to leverage the Restore application frequent sandbox seeding, as the process for non-production environments closely mirrors a production recovery. As a starting point for recovery tests, the [Restore Validation guide](https://www.grax.com/wp-content/uploads/2026/03/Restore-Validation-Best-Practices.pdf) is available as a sample runbook. CapStorm’s Customer Success team also supports customer working sessions to help test and validate the restore process. In the event of an emergency production recovery need, please [contact CapStorm](https://www.capstorm.com/contact-us/) by phone and by emailing . # Installation Instructions ## Installing CopyStorm (Windows) Installing CopyStorm takes a few seconds and does not touch any system files or registry values. You should not need an administrator. * Make sure your computer meets the prerequisites. * All required software is packaged in the download as long as a Win32 or Win64 version of the application is selected. * Download CopyStorm from * Create the directory “C:\Capstorm” * Unzip the downloaded zip file into the “C:\Capstorm” directory. A single directory named CopyStorm will be created. * Create the directory “C:\Capstorm\CopystormConfig”. * Copy the CapstormCustomerKey.txt file into the “C:\Capstorm\CopystormConfig” folder. * Click [here](/best-practices/customer-key-file-management) for more information about the CapstormCustomerKey.txt file. * Go to the “C:\Capstorm\CopyStorm” directory and: * Double click on CopyStorm.bat ## Installing CopyStorm (Linux/Mac) Installing CopyStorm takes a few seconds and does not touch any system files. You should not need root permissions. * Make sure your computer meets the prerequisites. * Java 11 (or above) must be installed. * Download CopyStorm from * Create the directory “/opt/capstorm” * Unzip the downloaded zip file into the “/opt/capstorm” directory. A single directory named CopyStorm will be created. * Create the directory “/opt/capstorm/CopystormConfig”. * Copy the CapstormCustomerKey.txt file into the “/opt/capstorm/CopystormConfig” folder. * Click [here](/best-practices/customer-key-file-management) for more information about the CapstormCustomerKey.txt file. * Go to the “/opt/capstorm/CopyStorm” directory and: * Run the command “sh CopyStorm.sh” (or make the launch script executable). ## Installing CopyStorm (Docker) CopyStorm can be installed in a Docker container by following instructions available here: * After creating the image and starting the container, it will run CopyStorm — backing up Salesforce to the configured relational database. When the backup finishes (successfully or with an error) the container will terminate. ## Salesforce Requirements * The API package must be enabled. * Enterprise, Unlimited, and Developer Editions have this feature enabled by default. * A Salesforce login that can read data and data setup information. ## CopyStorm Requirements CopyStorm is supported on: * Windows * Linux: * RHEL, SUSE, Ubuntu * MacOS * VMWare hosted Windows, Linux, or MacOS CopyStorm requires a Java 11 JRE or higher. Downloads indicating Windows 32bit or 64bit include the Java JRE, other systems require that a JRE be installed. Runtime requirements for the CopyStorm application can vary between installs – if the installation will only run one CopyStorm job at a time, a good starting point is: * 1 CPU * 2 GB RAM * 10 GB of free disk space * CopyStorm stores large objects (e.g. Attachments) on-disk while copying to minimize RAM usage. * A graphics head that can be used to run the CopyStorm configuration UI * On linux, X11 forwarding can work but is known to be extremely sluggish. * A desktop environment that can be remotely accessed gives the best performance. # Salesforce Job Logging Installing the CopyStorm Dashboard takes a few minutes and does not touch any system files or registry values. You should not need an administrator. * Ensure that you have the CopyStorm/Restore installed. * You will need version 10.54.2 or newer. * Contact CapStorm to request the package.- Open CopyStorm/Restore. * Open the “File” menu in the top left corner of the application UI, and select the “Load Package” option. * ![](/files/HLXiSUiaH1PLiwuzt4HG) * Open the downloaded package with the file selector dialog. * Navigate to the “Meta Data” tab, and then click the “Deploy” button along the bottom of the UI. * ![](/files/Oa2zu9JGw1MV2hqL2TRs) * This will open a new pop-up dialog titled “Deploying Metadata To Salesforce” * Click the “Deploy” button along the bottom of the dialog * ![](/files/YuGsNQeQdd3Tto6o51EP) * This will open the “Deploy Log” tab of the dialog where logging messages of the deployment process will be displayed. When the deployment has finished successfully, a “DEPLOYMENT SUCCEEDED” message will appear: * ![](/files/4jrMOpJQwG074WjvxcG9) * If a failure message appears instead, please contact [CapStorm Support](https://www.capstorm.com/contact-us/) ## Assigning the Salesforce Job Logging Permission Set After deploying the package, the Salesforce Job Logging Permission Set will need to be assigned to the Salesforce User provided to the CopyStorm application (specified on the “Main” tab of the CopyStorm application). The Permission Set was deployed as part of the deployment process above. * Login to the Salesforce Org where the package was deployed, and assign the Permission Set to the appropriate user. * The Permission Set is called “CapStorm\_CopyStorm\_Job\_Logging” ## Configuring CopyStorm for Salesforce Job Logging After the Permission Set has been assigned, CopyStorm will need to be configured to log job data to Salesforce. * Ensure that you have the CopyStorm installed. * You will need version 10.54.2 or newer. * Open CopyStorm, and navigate to the “Configuration” tab * Change the “Configuration Level” dropdown value to “Advanced”, then navigate to the “Logging” section * ![](/files/2kre5oMd1Dnl7VfGgEZB) * Change the “SFDC Logging” dropdown value to “On” * ![](/files/FFzDMMnUsCqwhilFcQQG) * Save the CopyStorm configuration. At this point, CopyStorm is now configured to log job data to Salesforce. # Best Practices # CopyStorm Directory Structure Upgrading CopyStorm can be a simple 30 second task if installed using a specific directory configuration. The directory structure we recommend for a CopyStorm installation looks like the following: * *CopyStorm* * Created by unzipping the CopyStorm download (see the Installation Instructions). * *CapstormConfig* * *CapstormCustomerKey.txt* * Capstorm Customer Key file used for licensing. * *ProductionA* * *ProductionA.copyStorm* * CopyStorm configuration file for backing up the ProductionA Salesforce. * *ProductionALessFrequent.copyStorm* * CopyStorm configuration file for backing up the ProductionA Salesforce. This configuration file may backup Salesforce tables that are more expensive to backup (examples: Attachment, ContentVersion). * *ProductionA.copyStormMedic* * CopyStorm/Medic configuration file for maintaining the ProductionA Salesforce backup. * *config* * XML configuration files used by CopyStorm when backing up the ProductionA Salesforce. * *ProductionB* * *ProductionB.copyStorm* * CopyStorm configuration file for backing up the ProductionB Salesforce. * *config* * XML configuration files used by CopyStorm when backing up the ProductionB Salesforce. In other words: * Create a directory next to the CopyStorm installation called CopyStormConfig to hold all of your site specific configurations and customizations. * Create a separate subdirectory under CopyStormConfig for each SalesForce instance to be backed up. If you are managing your own job scheduling rather than using CopyStorm/Director then the BAT or sh files used to run your scheduled jobs should be stored with their respective CopyStorm configuration files. If you are using CopyStorm/Director the CopyStormConfig directory works perfectly for configuring CopyStorm/Director. ## Upgrading CopyStorm to a new version If CopyStorm is installed using the directory structure detailed above, this process will upgrade the installation to a new version: 1. Remove or rename the CopyStorm directory. 2. Unzip the new CopyStorm release in the same location as the old CopyStorm directory. This process should end with a replacement CopyStorm directory containing an upgraded version of the CopyStorm software. # CopyStorm Configuration Files Many sites will decide to backup their Salesforce using multiple CopyStorm jobs. Why? * A site will want the data in specific tables in the CopyStorm database to be within minutes of the data in Salesforce. These tables in CopyStorm are frequently used for analytics or operational reporting. * Some tables in Salesforce simply take longer to backup and are not important to keep within minutes of Salesforce. Common culprits include Attachment, ContentVersion, and Knowledge Article version tables (each of these tables contain base64 columns and take a single API call to read a record). If you are not content with a single CopyStorm job then this article describes a few best practices for using multiple CopyStorm jobs. ## A Common Setup Strategy A common setup strategy involves dividing the tables in Salesforce into three separate buckets. 1. Tables to backup frequently. 2. Tables that are expensive to backup and can be backed up less frequently. 3. Metadata. For this setup, we create three separate CopyStorm configuration files and schedule them to run a different times and frequencies. ### Frequent Configuration File This configuration file is will be run frequently and backs up tables we want to keep in close agreement with Salesforce. Rather than specify what tables are in the frequent category, we specify the tables to exclude. Many sites will run this backup every 15 to 30 minutes. ![](/files/KWuJDjGPM9rPI4FXMMWQ) ### Less Frequent Configuration File This configuration file will be run infrequently and backs up tables that **do not** need to be kept within a few minutes of Salesforce. Use the “Include List” CopyStorm feature to specify the tables to back up. Many sites run this backup every few hours. ![](/files/D9NvavMKCNYwbRvTm6fs) ### Metadata Configuration File This configuration file will be run infrequently and backs up just Salesforce metadata. Use the “Include List” CopyStorm feature to tell CopyStorm to backup up just a single table — in this case the table specified is not in Salesforce. ![](/files/wWJiGKtlv2FSmhVWcq2K) ## Next Steps Once multiple CopyStorm configuration files have been created, jobs need to be created and scheduled. * Store the CopyStorm configuration files using [Capstorm’s Directory Structure Best Practices](/best-practices/copystorm-directory-structure). * Create a script for running each job using the [CopyStorm Batch File tab](/reference/batch-file-tab). * [Schedule each job](/frequently-asked-questions/installation/how-to-run-copystorm-in-batch-mode). ## Advanced Strategies There is no functional limit to the number of CopyStorm jobs you can use to backup Salesforce and some sites will use four or more jobs — usually to support keeping key CopyStorm tables within minutes of Salesforce. The choice is really yours. Here are few suggestions when running many jobs. * Try to avoid backing up the same table in multiple jobs. Though this will not damage the integrity of the backup it will often generate annoying messages. * Consider using a “super scheduler” tool like [CopyStorm/Director](https://learn.capstorm.com/copystorm-director/) to manage scheduling and logs. In general a super-scheduler make managing a lot of backup jobs and their logs much simpler. * Be aware that a single Salesforce Login Name can have at most 20 concurrent active queries. If you have more than 20, Salesforce may silently kill one of your sessions. Though CopyStorm will recover from this unfriendly Salesforce action automatically, many applications will not behave so nicely. # Optimizing CopyStorm Performance While achieving CopyStorm performance can vary from company to company, the principles and practices in this article should help you optimize CopyStorm for your organization. ## Common Case: Optimize Backup Performance In most cases a company will want to optimize the rate at which records are backed up from Salesforce. In this case there are typically only a few parameters that typically yield performance improvements which: * Increase the rate at which data arrives from Salesforce. * Increase the rate at which your database can accept changes. The best way to improve the rate at which data arrives from Salesforce is to: * Increase the number of tables CopyStorm backs up at the same time. This value is controlled by a parameter called “# Salesforce Threads” under “Salesforce API” on the “Configuration” tab. Most companies will see optimal performance using a value of 3 to 10. The optimal value for your company will likely depend on the rate at which your database will accept data. ![](/files/nNCFuENKVOREn7mmikft) People often ask “Why not use the BULK API since it is the fastest approach?” The short answer is: * Though the execution time reported by the BULK API is short, the amount of time a human has to wait (wall time) for a copy job to complete is longer than using multiple SOAP or REST threads. Ten SOAP threads will beat ten BULK jobs almost 100% of the time. We suspect that this is because BULK jobs run at a much lower priority in Salesforce than SOAP or REST requests. ## Special Situation: Optimize Memory Usage There are two common cases where memory usage is a problem: ### Large Attachments Because the Salesforce SOAP API returns Attachments inline the memory requirements for large SOAP attachments will be almost four times the size of the attachment reported by Salesforce. There are two primary ways to solve this problem: 1. [Increase the memory allowed to CopyStorm](/frequently-asked-questions/problems-solutions/out-of-memory-java-heap-space-error). 2. If using Oracle or SQL/Server switch to the REST API. In this case large Attachments will no longer be kept in memory and will be streamed directly to the database. The REST API helps slightly for other databases but not dramatically, as they require large objects to be kept in memory ### Old Equipment If you are forced to work on equipment that does not have much memory (e.g. an old 32-bit 2GB Windows box) then: 1. Huge attachments may not be possible (unless using Oracle or SQL/Server). 2. CopyStorm memory use can be limited at the cost of more Salesforce API calls by reducing the “Salesforce Batch Size” parameter. CopyStorm typically has very modest compute requirements — if you find yourself in a situation requiring more memory or compute than available Capstorm generally recommends upgrading hardware (a standard desktop has more than enough resources to run CopyStorm). ## Special Situation: Minimize Lock Contention If your CopyStorm database is heavily used for business analytics, operational reporting, or other purposes while a CopyStorm backup is running you may experience database performance issues due to lock contentions. The easiest way to solve this problem is to reduce the number of records sent to the database in a single transaction: ![](/files/NbWTaWJCLjVQfD8RYHBZ) # Microsoft SQL/Server Database Setup The approach taken when setting up a CopyStorm/SQL Server database may go a long way in reducing long-term ownership costs. There are two primary recommendations that should be followed when configuring a CopyStorm/SQL Server database. ## Use a Service Account The most common “mistake” made when setting up a SQL/Server database for CopyStorm is to use a Domain User along with a user specific schema for storing Salesforce data. For example, 1. The database administrator creates a database named “salesforce”. 2. User CAPSTORM\greg runs CopyStorm using integrated security (e.g. a domain user login) and creates all database tables in the \[CAPSTORM\greg] schema. 3. User CAPSTORM\noah wants to use the CopyStorm database but quickly becomes irritated because: * His default schema is \[CAPSTORM\noah]. * All SQL table references have to include \[CAPSTORM\greg]. Though a database administrator can get around this problem by making the default schema be \[CAPSTORM\greg] for all CopyStorm database users, this is not a great idea. The core problem with this process is that the ownership of shared Salesforce backup data is tied to a real person — when that person wins the lottery or moves to a new position someone else will need to own the data. **Core Principle:** The owner of a SQL/Server Salesforce database **should not** be a Windows Domain Account. Capstorm considers the following to be the best practice for creating and populating a CopyStorm database in SQL/Server: 1. Create a new SQL/Server database. * The only purpose of this database is to hold CopyStorm created backups of Salesforce. 2. Create a non-domain SQL/Server user. Commonly used names include “salesforce”, “copystorm”, or “capstorm”. * Give this user ownership of the dbo schema in the CopyStorm database. * Make the default schema “dbo” in the CopyStorm database. 3. Always run CopyStorm using the non-domain user. 4. Create a group with read-only access to the salesforce database with a default schema of “dbo”. 5. When a domain user needs access to the SQL/Server Salesforce database, make them of a member of the group created in the previous step. If your organization is smaller, the only essential best practice steps are 1-3. If your organization chooses to use domain accounts for all SQL/Server authentication that’s OK, with the caveat that the CopyStorm database will require permission updates when the person tied to the domain account decides to retire. ## Enable Forced Parameterization Performance of CapStorm applications can be greatly impacted by the SQL Server query planner. By default, the “parameterization” parameter is set to “SIMPLE” – which can cause common CapStorm queries to not be fully optimized by the SQL Server query planner. It is recommended that you set the “parameterization” parameter to “FORCED” – this will enable SQL Server to better optimize query plans for common CapStorm application queries. You can find out if parameterization is forced by running this query: * `SELECT name, is_parameterization_forced FROM sys.databases WHERE name=DB_NAME()` And to enable forced parameterization, you can run this query: * `ALTER DATABASE CURRENT SET PARAMETERIZATION FORCED` # Oracle Database Setup This article focuses on specific technical decisions that need to be made when setting up an Oracle database for CopyStorm. If you need more general Oracle help then grab a DBA — Oracle shops always have them. ## Use an Oracle Character Set Compatible with Salesforce In short, use AL32UTF8. Salesforce uses the full extended UTF-8 character set (up to 4 bytes per character) and an equivalent character set should be used in Oracle. Do not use UTF8 even though your backup will almost always work since, despite its name, the Oracle UTF8 set is not a full implementation of the UTF8 standard. ## Enable Oracle EXTENDED Characters By default CopyStorm will use VARCHAR2 columns until the declaration of a column exceeds 4000 bytes. After 4000 bytes CopyStorm will use a CLOB column type. If you have a version of Oracle that supports the *max\_string\_size=EXTENDE* D property, extending VARCHAR2 lengths to 32767 bytes, you should consider forcing CopyStorm to use this feature. Unfortunately, since automatically detecting EXTENDED support cannot be done by a regular Oracle user but requires nearly the same permissions as SYSTEM, CopyStorm must be configured to use the feature by following the directions in this article: * [How to Enabled Oracle’s EXTENDED Options for VARCHAR2 Columns](/reference/configuration-files/how-to-enable-oracle-extended-varchar2-option) ## The 30 Character Name Limit Is Not a Problem CopyStorm automatically detects the maximum length for names at run-time and will use 30 or 128 long names depending on what your Oracle version supports. In all cases the mappings between Salesforce table and field names are kept in tables in the CopyStorm created database. See this article for the details. * [Where Does CopyStorm Keep Salesforce Metadata?](/frequently-asked-questions/how-does-copystorm-work/where-does-copystorm-keep-salesforce-metadata) # MySQL Database Setup The simplest way to setup a new database for CopyStorm in MySQL is: * `CREATE SCHEMA copystorm` However, it is important that your database server be configured to support the full set of UTF-8 characters supported by Salesforce. Default MySQL configuration typically uses the latin1 character set, which omits UTF-8 characters like a smiley face with sunglasses (U+1F60E). If a non-included character appears in Salesforce, MySQL will refuse to accept it and report an error like: * `SQL state [HY000]; error code [1366]; Incorrect string value: ‘\xF0\x93\x83\xA2\xF0\x93…’` ## Make Sure MySQL Supports the Full UTF-8 Character Set You can tell if MySQL is already configured correctly by running the following command: * `SHOW VARIABLES LIKE ‘character_set_server’` If the current value of “character\_set\_server” is “utf8mb4” then your server is setup correctly. If not, here are the steps to correct the problem: * Open the mysqld.cnf file for your MySQL. This contains the default parameters for your server. * Look for the “character\_set\_server” setting. If you find it then change it to utf8mb4 and save the changes. If not, then add lines like the following to the “\[mysqld]” section: * character\_set\_server = utf8mb4 * character-set-server = utf8mb4 * Which line depends on the exact MySQL version and OS distribution you are using. * Restart MySQL. * Run SHOW VARIABLES to verify the change. ## Some Characters Look Wrong in SQL Workbench – Why? MySQL Workbench often uses a default character set of utf8. If a string containing 4-byte utf8 characters is viewed in Workbench then the 4 byte characters appear as question marks. To fix this behavior, set the character set for the session: * `SET CHARACTER SET utf8mb4;` If your workbench session still does not correctly display the characters then it may be a workbench problem. We have seen cases where Heidi (the MariaDB Workbench equivalent) and SQuirrel display the data correctly and Workbench does not. ## Other Good Advice ### Use a 64KB Page Size Instance page size is set (using the variable innodb\_page\_size) when you first create a MySQL instance. This can cause problems when Salesforce contains very wide tables whose records will not fit into a single page (by default 16KB), and cannot be changed without deleting and rebuilding the database. Capstorm always recommends using a 64KB page size when setting up a Capstorm MySQL database. ### Allow For a Large Log File MySQL’s maximum log file size needs to be around 4 times the size of the large object in MySQL. If this is not the case you may see a “Row Size Too Wide” error from MySQL. To change your log file size search for the “innodb\_log\_file\_size” in the MySQL documentation. ### Use Dynamic Tables to Avoid Problems With Many Large TEXT Columns The default InnoDB format, Antelope, stores the first 768 bytes of a TEXT column with the row data — the rest is stored elsewhere. Since Antelope is the default InnoDB format almost every MySQL instance will be using it. If at runtime a row has a lot of TEXT columns and many of them have more than 768 bytes, you may see a “Row Size Too Wide” error message from MySQL. An easy way to avoid this issue is to use the InnoDB Barracuda format and change tables to use ROW\_FORMAT=Dynamic. To change a database to use the Barracuda format: * `SET GLOBAL innodb_file_format = barracuda;` * `SET GLOBAL innodb_file_format_max = barracuda;` This will cause future tables to use the Barracuda format. To force a table to use a Dynamic row format: * `ALTER TABLE tableName ROW_FORMAT=Dynamic;` In addition, you can force all tables created by CopyStorm to be Dynamic Barracuda tables by adding the following lines to the config file “TableRuleRegistry.xml”: ```xml ``` # PostgreSQL Setup To get the most out of your PostgreSQL database there are a number of tasks you will want to do after you have installed the PostgreSQL Server ## Increase PostgreSQL Memory Allocations The default maximum memory allocated to a default PostgresQL is minuscule. Yes, this means that a stock PostgreSQL will run on a computer with less capability than a modern cell phone but it also means a lot of unnecessary disk swapping. There are a handful of parameters you will want to bump — all found in postgresql.conf. The following table lists a few key default PostgreSQL parameters and the parameters used in Capstorm’s labs. | Parameter | Default | Capstorm PostgreSQL Servers | | ---------------------- | ------- | --------------------------- | | shared\_buffers | 128MB | 4GB | | work\_mem | 4MB | 1GB | | maintenance\_work\_mem | 64MB | 1GB | | effective\_cache\_size | 4GB | 12GB | For additional help see: * # Formula Management By default, CopyStorm inserts/updates records in the Salesforce backup database when the corresponding record timestamps change in Salesforce. When Salesforce formula fields are involved, a field value may change without the timestamp on a record changing. This article describes several mechanisms provided by CopyStorm to keep database columns in sync with the corresponding formula columns in Salesforce. ## Two Types of Formula Fields Salesforce supports two classifications of formula fields: * Row-based formulas * Determine their value based on the values of other columns/fields in their row/record. * Values for this type of formula change only when another field in the same record changes. * Variant formulas * Determine their value based on data found in other records (data stored anywhere but the same row/record). * This classification of fields includes: * Summary fields. * Fields computed from a changing value (e.g. “CloseDate – TODAY()”) ### Row-Based Formula Fields Maintenance of Row-Based Formula fields is fairly simple because there are only three conditions where the value of the formula field can change: 1. A new formula field is created. * In this case all records modified AFTER the formula field is created will automatically be updated in CopyStorm. * Records whose last modification date is earlier than the formula creation date must be manually refreshed. 2. The definition of an existing formula field is modified. * In this case all records modified AFTER the formula field is created will automatically be updated in CopyStorm. * Records whose last modification date is earlier than the formula creation date must be manually refreshed. 3. The data used to compute the formula field changes. * Since the data used to compute a Row-Based Formula field is in the same row as the formula, this will change the modification timestamp on the record and changes will automatically be reflected in CopyStorm. The following scenario illustrates the problem to solve: 1. Suppose: * The Account table has 100 existing rows and no formulas. * CopyStorm has just copied all 100 rows to a database. 2. A programmer adds a new formula column named “age\_\_c” using the formula “1+2”. 3. CopyStorm runs and it: * Creates a new database column named “age\_\_c”. * Makes no changes to the database (because no records have been updated there are no modification dates later than the previous CopyStorm backup). * The age\_\_c column defaults to NULL for all records in the CopyStorm database. 4. A user makes a change to the account ACME. 5. CopyStorm runs and it: * Detects that the ACME record has changed and updates the database. * Sets the age\_\_c value for ACME to “3”. * No other Accounts are updated and retain NULL for age\_\_c (because none have modification timestamps later than the previous CopyStorm backup). 6. A programmer modifies the age\_\_c column and changes the formula to “1+2+3”. 7. A user makes a change to the account BIOSPHERE. 8. CopyStorm runs and it: * Detects that the BIOSPHERE record has changed and updates the database. * The BIOSPHERE record has a value of “6” for the age\_\_c column. * The ACME record retains the value of “3” for the age\_\_c column. * All other Accounts retain the value of NULL for the age\_\_c column. Our recommended best practice is to force an update on a Row-Based Formula column any time: * A Row-Based Formula column is created. * A Row-Based Formula column is modified. Forcing a column update only takes a few minutes (in the GUI or as a batch job) using the [CopyStorm/Medic RefreshColumns](/reference/copystorm-medic/refresh-columns) tool. Alternatively, the [CopyStorm/Medic RefreshFormulas](/reference/copystorm-medic/refresh-formulas) tool can be used to refresh every formula in a table. Another solution is to truncate or drop the corresponding database table and let CopyStorm rebuild it during the next backup. The best practice for Row-Based Formula columns is to run [CopyStorm/Medic RefreshColumns](/reference/copystorm-medic/refresh-columns) on the column when it is created and any time the formula definition changes. ### Variant Formula Fields A Variant Formula field has a value that changes even when the data in its corresponding row has not changed. These changes may occur every minute, hour, month, or irregularly. Examples of Variant Formula fields include: * The number of minutes a support case has been in an open state. * The number of hours it took for an account rep to open a new lead. * The number of months since a customer was last contacted. * The total amount of bookings from an account in the current year. In every case, keeping a CopyStorm database in sync with Variant Formula fields often requires a business process decision to be made for each field. The best practice is to: * Divide Variant Formula fields into different buckets based on how frequently their value in CopyStorm should be synced with Salesforce. * Create a CopyStorm/Medic configuration file for each bucket of formula fields. Each configuration file should be configured to update specific columns using the [CopyStorm/Medic RefreshColumns](/reference/copystorm-medic/refresh-columns) tool. * Create a scheduled job for each CopyStorm/Medic configuration file. If you have a lot of CopyStorm/Medic jobs to run due to having a complex set of Variant Formula fields then you may want to consider [CopyStorm/Director](https://learn.capstorm.com/copystorm-director/). CopyStorm/Director is designed to manage and track the scheduling and status of many CopyStorm and CopyStorm/Medic jobs. # Deleted Record Management Though CopyStorm will faithfully back up records that have been deleted in Salesforce, there are a few cases where Salesforce will delete records without the deletion being visible to CopyStorm. These situations occur when: * A *hard delete* is performed in Salesforce. * A Salesforce table does not track deleted records (this occurs in system tables). ## Best Practices for Hard Deleted Records When a Salesforce record is deleted, Salesforce marks the record as deleted by setting the “isDeleted” field to true and by putting the record in the recycle bin. When CopyStorm runs it retrieves all updated records — including those which have been marked as deleted. This is used to record record deletion in the CopyStorm database, but there are a few cases where a record is deleted and not put in the recycle bin. This is called a hard delete. When this happens, the normal CopyStorm update flow will never determine that the record has been deleted. There are four primary conditions where hard deletes occur: * A user manually empties a recycle bin. * A record has been in the recycle bin for a long time (weeks) and Salesforce automatically deletes it from the recycle bin. * Two records are merged. * Salesforce deletes a record without using the recycle bin. This does not occur on most tables, but this article identifies the handful of tables with this issue. To solve this problem, the “isDeleted” field needs to be updated to reflect records that have been hard deleted in Salesforce. The [CopyStorm/Medic fixIsDeleted](/reference/copystorm-medic/fix-isdeleted) tool has been designed to do this exact task. How often (and if) your site resolves hard deletes is up to you. If you would like the CopyStorm backup to reflect hard deletes then the following list may help — it contains known tables that can contain hard deletes in a out-of-the-box Salesforce instance: | **Table Name** | **Common Impact** | **Description** | | ---------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | OpportunityLineItem | High | The operation of syncing an opportunity with a quote does a hard delete on existing line items. | | OpportunityContactRole | High | When team assignments change hard deletes from Salesforce can happen. | | .\*History | Low | Older history records can be automatically hard deleted by Salesforce. | | .\*Share | Low | When sharing rules are modified, Saleforce hard deletes share records. | | .\*Feed | Low | Feed records (from Chatter) are sometimes deleted for reasons that are not clear. We suspect this occurs when a chatter feed is deleted. | Because hard deletes can be caused by a site’s business practices (or by specific user behavior), best practices are often site specific. This is Capstorm’s recommendation: * If your site uses Salesforce quotes, then run a daily CopyStorm/Medic job to fix hard delete problems within the OpportunityLineItem table. * Each weekend, run CopyStorm/Medic to fix hard delete problems on all custom tables and core Salesforce tables. Core Salesforce tables are listed in the sample CopyStorm/Medic file found here. {% file src="/files/wjQhLGhZGuV4Ivk9ffIY" %} In practice, we have found that very few custom or core Salesforce tables need to be scanned. Over time we reduce the number of tables scanned based on problems reported by CopyStorm/Medic. ## Best Practices for Non-Tracked Records There are a number of system tables in Salesforce which do not track deletions or allow restoration from the Salesforce recycle bin. Tables that follow this behavior can be found in the CopyStorm database by running the following SQL query: * `select distinct tableName from copyForceTableField where tableName not in (select tableName FROM copyForceTableField where name=’IsDeleted’);` Most of the time a site will not care if these tables contain deleted records. If your site requires that deleted data be pruned, the best practice is to periodically drop or truncate these tables. Once dropped, the next run of CopyStorm will automatically repopulate the tables. # Field History Management In CopyStorm there is a lot of flexibility in how field history tracking is setup. Specifically: * Field history tracking is available for all fields in all tables. * The number of days fields history is kept is controllable on a per table basis. * Field history can be kept forever. Though it is not difficult to set up field history rules, there are a handful of questions that need to be answered. * Do I want to control what tables are backed up using a Whitelist or Blacklist approach? * By default, how many days should field history records age before they are automatically deleted by CopyStorm? * Are there any tables whose field history records I want to keep longer (or shorter) than the default? The rest of this document provides examples for both a Blacklist and Whitelist approach. ## Example: Blacklist Field History Based Tracking In a blacklist approach to field history the tables that SHOULD NOT be tracked must be specified. In addition to deciding on what tables to be excluded there are two other decisions to make: * By default, how many days should field history record be kept before being deleted from CopyStorm? * Are there any tables whose field history records should be kept longer (or shorter) than the default? For the following example: * By default, keep field history records for 90 days. * Exclude Salesforce History, Feed, and EventLog records from field history tracking. * Keep Account, Contact, Case, and Email fields history changes for 800 days. ![](/files/u645kk42gHgxOHAqs86E) ![](/files/3kJYiSH5G4gwy78maYuU) ![](/files/enHQNEgkvDV9BvijYW9o) ![](/files/V06Lb2MVwLAaaIuPD3pI) ## Example: Whitelist Field History Based Tracking In a whitelist approach to field history tracking, all tables to be tracked are explicitly specified. The advantage is precision while the disadvantage is that any new tables added to Salesforce will not participate in field history tracking by default. In addition to deciding on what tables to include there are two other decisions to make: * By default, how many days should field history record be kept before being deleted from CopyStorm? * Are there any tables whose field history records should be kept longer (or shorter) than the default? For the following example: * By default, keep field history records for 90 days. * Include a lot of tables. * Keep Account, Contact, Case, and Email fields history changes for 800 days. ![](/files/rNh4y3ns62rioTM19bdn) ![](/files/sXS5M2gYOPPMQMqn5J5h) # Data Retention Policy CopyStorm provides a robust data retention facility that allows Salesforce records to be recoverable at virtually any point in time, and provides a lot of flexibility in establishing a data retention policy. What is a data retention policy? A data retention policy is a set of practices that retain historical corporate data for a specified period of time and, in the case of databases, allows historical data to be restored. A good data retention policy supports tasks like “Restore this data to how it was last Wednesday”. There are two ways to configure a data retention policy within CopyStorm: * Using CopyStorm’s built-in “Default Point-In-Time” Snapshot Policy to capture virtually any record change that may be needed for restoration later. * Creating a custom data retention Snapshot data source, capturing a targeted subset of data for later restoration. The rest of this document provides information useful for setting up CopyStorm’s built-in “Default Point-In-Time” Snapshot Policy. For more comprehensive details on how to configure a custom Snapshot Policy to suit specific data retention needs, please see the documentation for [CopyStorm’s Snapshot tab](/reference/archive-tab). ## Configuring CopyStorm’s Default Point-In-Time Snapshot Policy CopyStorm’s built-in “Default Point-In-Time” Snapshot Policy is preconfigured to record all data changes that may be required for recovery. To enable CopyStorm’s “Default Point-In-Time” Snapshot Policy: 1. Go to CopyStorm’s Snapshot tab. 2. Click on the “Edit” link for the Default Point-In-Time Snapshot Policy. 3. Select the “Active” checkbox. * This will enable the snapshot, after being enabled CopyStorm will automatically capture record changes in the Default Point-In-Time data source. 4. Set the # Days to Retain record changes. * All record changes will be stored in the database for the specified number of days. 5. Save changes. ![](/files/WmIsMOOD1JJFwzk0xGi2) ![](/files/EIeDAetMqQs0CxeySqmi) # Setup OAuth Application Access The purpose of this page is to document how to configure CopyStorm desktop apps to require login via OAuth 2.0 identity provider. This feature updates CapStorm applications to require a login prior to accessing the application GUI – this feature is separate from features enabling OAuth for Salesforce or database login. OAuth login for CapStorm desktop applications is only supported on Windows. ## Installation * Download the plugin. * OAuth application authentication requires a plugin to be installed into CapStorm desktop applications. * Contact your CapStorm account representative to obtain the plugin zip file. * Install the plugin. * Extract the OAuth2 plugin zip file to the *plugin* folder in the CapStorm application installation directory. * The *plugin* is a sub-folder of the main application installation directory – i.e. it is next to the “config” folder and “CopyStorm.bat” file. ## Configuration OAuth settings are stored in a file named *AuthParams.xml* in the CopyStorm config/ directory. The specification for this file is: ``` ``` * Request: Parameters to configure OAuth requests. * url: the IdP’s authorization endpoint. * Param elements under the Request element are added to the OAuth HTTP request. * Validate: Parameters to configure OAuth validation requests. * url: the IdP’s token validation endpoint. * Param elements under the Validate element are added to the token validation HTTP request. * Be sure to update the following parameters inside the *AuthParams.xml*: * **client\_id** * **client\_secret** * **redirect\_uri** Make sure the **redirect\_uri** matches the one registered in the OAuth IdP. The CapStorm OAuth plugin requires token validation – if the OAuth request or token validation fails, then the application will not launch. ## Using OAuth to access CopyStorm When this feature is configured, CopyStorm will perform the following steps before displaying the application UI: * Check for a valid AuthParams.xml file. * Launch a browser, navigating to the IdP login page. * Wait for the authorization code callback. * Validate the authorization code using the token endpoint. * Continue startup only if authentication succeeds. If the OAuth login fails, the token request fails, the token fails validation, or the AuthParams.xml file is misconfigured – CopyStorm will not launch. Instead of launching the application, CopyStorm will either display an error window or write the error message to STDOUT / STDERR. If you do not see an error message, try launching CopyStorm from a CMD window as “CopyStorm.bat -debug”. ## Getting Help If assistance enabling this feature is needed please reach out to your CapStorm account representative or [CapStorm support](https://www.capstorm.com/contact-us/). # Customer Key File Management Capstorm licensing is implemented using a unique Customer Key. The purpose of a Customer Key is to uniquely identify a customer to the Capstorm licensing system, so the licensing system can implement policies unique to a customer. The Customer Key is associated with a single organization’s use of Capstorm software and is used to determine which applications and features may be used. ## Obtaining and Installing a Customer Key The only way to acquire a customer key is to contact your Account Executive or . They will provide you with a customer key contained in a file named: * CapstormCustomerKey.txt You have a number of choices when determining where to install the customer key file: ![](/files/sRaYZLV8WMAVeSxEQZFQ) ### Option 1: A Directory At the Same Level as Capstorm Products If you have followed [best practices](/best-practices/copystorm-directory-structure) and installed your Capstorm products in a directory structure like: * /opt/Capstorm/CopyStorm/ * /opt/Capstorm/CopyStormRestore/ * /opt/Capstorm/CopyStormMedic/ * /opt/Capstorm/CapstormConfig/ The key file can be stored at: * /opt/Capstorm/CapstormConfig/CapstormCustomerKey.txt Once installed, all Capstorm applications will find it and use it. ### Option 2: Capstorm Product Directory If the customer key file is stored in the root directory of any Capstorm application, the application will find it and use it when talking to the Capstorm license server. For example, if CopyStorm is installed at “/opt/Capstorm/CopyStorm”, the key file can be stored at: * /opt/Capstorm/CopyStorm/CapstormCustomerKey.txt ### Option 3: User Home Directory The key file may also be installed in the user’s home directory: * Windows: C:\Users\*Username\*\CapstormCustomerKey.txt * Linux: /home/*Username*/CapstormCustomerKey.txt # MySQL and CopyStorm Version Upgrades There are a number of quirks to MySQL when it comes to adding columns or altering columns in an existing table. The technique it employs, unlike other databases supported by Capstorm, can be quite time-consuming. To perform an *ALTER TABLE abc ADD COLUMN* command, MySQL: * Creates an empty shadow of the original table WITH the new column. * Copies the entire existing table to the shadow table. * This can take a long time for huge tables. * Replaces the existing table with the shadow table. This MySQL behavior can make it faster to rebuild a CopyStorm database when upgrading protocols rather than upgrading a MySQL database in place. ## How Do I Determine If CopyStorm Contains an Updated Protocol? CopyStorm version numbers contain three parts: * Major Version * Protocol Version * Minor Version For example, 7.34.1 is major version 7, Salesforce protocol 34, and minor version 1. A value of 7.31.1 would indicate the same feature set but using Salesforce protocol 31. ## What Happens When Salesforce Updates Protocols? When Salesforce updates protocols new columns and tables are almost always added (this happens 3 times a year, CopyStorm releases a new protocol update after each Summer protocol update). The first time you run CopyStorm with an updated protocol on your MySQL database new tables will be added (not a problem) and new columns may be added to existing tables (a problem for large tables). ## What Is the Best Practice? There is no hard and fast “best practice” but there are a few options that have various tradeoffs. If you are keeping deleted records forever or preserving deleted columns then your only option is to let MySQL upgrade the tables in place. Other approaches would cause you to lose data. If you do not keep deleted records and your organization can live with a table being offline for a small amount of time, the easiest option is to run CopyStorm once with the “Force Rebuild” option checked so tables can be fully rebuilt. This option will make CopyStorm drop and rebuild each table when backing up Salesforce, and can be found on the “Advanced” tab. This option should always be turned off when performing incremental backups — dropping and rebuilding each table each time a backup runs defeats the purpose of being incremental. If you do not keep deleted records but your organization wants the MySQL database down for the minimum amount of time, then: * Use the new version of CopyStorm to create a new database while continuing to run the older version of CopyStorm on the older database. * When the new version of CopyStorm is done backing up Salesforce to the new database, use it to replace the older database. Use the new version of CopyStorm going forward. # Reference # Main Tab The Main Tab is used to manage connection information and credentials for Salesforce and the target database. ![](/files/EAi4svX03hgEusOumpam) ## Copy Salesforce To Specifies the type of target database. Available database targets depend on the version of CopyStorm you have installed. ## Salesforce Connection Parameters These parameters are used when authenticating with Salesforce using a Username and Password. * For certificate based authentication, see the [Oauth 2.0 JWT Configuration reference](/frequently-asked-questions/usage/where-does-copystorm-store-passwords/how-to-login-as-a-connected-application). * For SSO-based authentication, see [How to Login to Salesforce using SSO](/frequently-asked-questions/problems-solutions/how-to-login-to-salesforce-using-sso). | **Parameter** | **Description** | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Instance Type |

Type of Salesforce instance being copied.

  • Production
  • Sandbox
  • Database.com
| | Username | A Salesforce user. This user must have API privileges. All data visible to this user will be copied to the CapStorm database. | | Password | The password for the Salesforce user. | | Security Token | Optional security token supplied by Salesforce. | | Alternate Host |

If your connection to Salesforce does not use one of the standard endpoint (login.salesforce.com or test.salesforce.com) then you must specify your endpoint as an alternate host. Example:

  • capstorm.my.salesforce.com
| | Advanced Parameters |

Launch this dialog to set:

  • Proxy information
  • Password storage model
  • Secondary Salesforce connection parameters
| ## Salesforce Advanced Parameters ![](/files/1kk031qUBDyc2LYytUjW) | Parameter | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Credential Type | Technique used to find the Salesforce password. The default, *Username and Password*, stores encrypted credentials in a project’s .copyStorm configuration file. To login to Salesforce using the JWT Oauth flow, change this to “Oauth 2.0 JWT” and see the [JWT Configuration Reference](/frequently-asked-questions/usage/where-does-copystorm-store-passwords/how-to-login-as-a-connected-application). To login using an SFDX registered instance, change this to “SFDX Username or Alias” and see the [SFDX Username Configuration Reference](/frequently-asked-questions/usage/where-does-copystorm-store-passwords/how-to-login-to-salesforce-using-sfdx). | | Salesforce Batch Size | Default number of records to read from Salesforce in a single batch. The default, 2000, is the maximum supported by Salesforce and should not be changed unless the host machine has severe memory limitations. Decreasing this value will increase the number of API calls to Salesforce. | | Max Timeout | Number of seconds to wait for a response from Salesforce. | | Describe Batch Size | Number of table descriptors to read from Salesforce in a single batch. | | Apache/HttpClient | If checked then the Apache/HttpClient will be used to communicate with Salesforce instead of the technique built into the WSC library. It is unusual to need the Apache/HttpClient. | | Proxy Host | Optional name or IP address of a proxy host | | Proxy Port | Optional port on which a proxy host is listening. | | Proxy User | Optional user name when a proxy requires it. | | Proxy Password | Optional password when a proxy requires it. | ## Database Connection Parameters | **Parameter** | **Description** | | ------------------------- | -------------------------------------------------------------------------------------- | | Connection String | JDBC connection string required to create a CopyStorm database. | | Connection String Builder | Clicking on this button launches a dialog that helps create a valid connection string. | | Username | CopyStorm database username. | | Password | CopyStorm database password. | | Schema | Default schema to use. Note that this parameter is not supported by all databases. | ## Database-Specific Configuration * [MySQL](/reference/main-tab/mysql-connection) * [Oracle](/reference/main-tab/oracle-connection) * [PostgreSQL](/reference/main-tab/postgresql-connection) * [SQL/Server](/reference/main-tab/sqlserver-connection) * [Snowflake](/reference/main-tab/snowflake-connection) # SQL/Server Connection Prior to running CopyStorm for the first time, create a fresh SQL/Server database. ## SQL/Server Connection String The only difficult part when copying Salesforce to SQL/Server is constructing the connection string. Clicking on the Connection String Builder will open a form that, if you are running SQL/Server using a standard configuration, will construct a connection string for you. The connection string can be verified by pressing the “Test SQLServer Connection” button. The general form of a SQL/Server connection string is: * `[server-name[\instance-name][:port-number]][;property=value]*` If you would like to see more information regarding SQL/Server connection strings or are not familiar with Microsoft’s type 4 JDBC driver, these resources may help: * * # MySQL Connection Capstorm includes a MySQL driver from the [MariaDB project](https://mariadb.com/) to communicate with both Oracle’s MySQL product and with MariaDB. Since MariaDB is provided by the original authors of MySQL and strives to maintain compatibility with Oracle’s MySQL, the driver packaged with CopyStorm is probably the only driver you will need. However, if you are encountering compatibility issues or other problems with MariaDB’s MySQL driver, you can download and install Oracle’s MySQL driver by hand. CopyStorm does not package this driver with the product due to Oracle’s licensing policy for ISV’s, which makes it difficult to package the Oracle MySQL JDBC driver with a product. ## How to Install Oracle’s MySQL JDBC Driver (not recommended for most cases) To install the Oracle MySQL JDBC driver: * Delete the file lib/mariadb-java-client-\*.jar from the CopyStorm installation directory. * Restart CopyStorm. * Choose MySQL as the database type. * CopyStorm will ask you if it’s OK to download and install Oracle’s JDBC driver. If you would prefer to install Oracle’s JDBC driver by hand: * Go to Oracle’s download page * * Download and unzip the driver. * Copy the jar file to CopyStorm’s lib directory. * Restart CopyStorm ## Common Problem – TCP/IP is not enabled All MySQL JDBC drivers communicate with MySQL using TCP/IP. If CopyStorm is having trouble connecting to the MySQL database, you may not have TCP/IP enabled. If so, there are a number of resources providing instructions on enabling this feature (the manual for your version of MySQL, running a web search for “MySQL enable TCP”, etc…). ## MySQL Connection String The only complicated configuration required in the CopyStorm Main tab is the MySQL connection string. Clicking on the Connection String Builder button will open a form that will construct the connection string for you, given database information. The connection string can be verified by pressing the “Test MySQL Connection” button. The format of a MySQL connection string is: * `//[host][:port]/[database-name]` Example: * `//localhost/MyCopyStormDatabase` # Oracle Connection ## Initial Step Prior to running CopyStorm for the first time, create a fresh Oracle database. ## Oracle Connection String The only complicated configuration required in the CopyStorm Main tab is the Oracle connection string. Clicking on the Connection String Builder button will open a form that will construct the connection string for you, given database information. You will likely need one of these forms: * `myDomain.com:1521:orcl` * `myDomain.com:1521:xe` * `localhost:1521` The connection string can be verified by pressing the “Test Oracle Connection” button. The following is a screen shot of a working Oracle setup on one of our test computers: ![](/files/iTibexiQhc2YMgjF6cnE) # PostgreSQL Connection ## Initial Steps Prior to running CopyStorm for the first time, create a fresh PostgreSQL database. ## Connection String The only complicated configuration required in the CopyStorm Main tab is the PostgreSQL connection string. Clicking on the Connection String Builder button will open a form that will construct the connection string for you, given database information. The general form of a PostgreSQL connection string is: * `[//host[:port]][/database-name]` Example: * `//localhost/CopystormBackupDatabase` By default, a PostgreSQL connection will use the public schema. This can be overridden by entering a schema name. The following is a screen shot of a working PostgreSQL setup on one of our test computers: ![](/files/BraowXIQjk6UhV8GC3IU) # Snowflake Connection ## Initial Steps Before running CopyStorm for the first time, create a dedicated Snowflake database for your Salesforce data. ## Connection String Compared to the other Database types, Snowflake’s connection string is more complex and carries more information. For this reason, it is recommended to use the Connection String Builder to create a valid Connection String. ![](/files/DVhXYNNfEanUySvUMXCh) ## Connection String Builder After clicking on the “Connection String Builder” button, a dialog menu will open, as seen below. ![](/files/olVkb31FPTHH9TEbnU6f) | Parameter | Description | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Server Name | The IP Address, URL, or server name of the snowflake instance. Port number is optional. | | Database Name | The name of the Snowflake database that CopyStorm will populate with Salesforce data. | | Private Key File | The file path to the Snowflake Private key. For more information on Private Keys and Snowflake’s key-pair authentication, refer to [Snowflake Documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth). | | Key File Password | If your Private key was configured to require a password for use in authentication, enter it here. If your Snowflake Private key was configured as to not require a password, this field can be left blank. | | Warehouse (optional) | The name of a Warehouse in Snowflake that you wish to use. For more information on Snowflake Warehouses, refer to [Snowflake Documentation](https://docs.snowflake.com/en/user-guide/warehouses-overview). | | Proxy Host (optional) | The IP address, or URL of a proxy used to connect to a Snowflake instance. | | Proxy Port (optional) | The port number associated with a proxy used to connect to a Snowflake instance. | | Proxy User (optional) | The username to authenticate a request with a proxy used to connect to a Snowflake instance. | | Proxy Password (optional) | The password used to authenticate a request to a proxy used to connect to a Snowflake instance | | Proxy Protocol (optional) | The protocol by which a connection will be made to a proxy used to connect to a Snowflake instance. Select either HTTP, or HTTPS. | # Configuration Tab The Configuration Tab contains options to modify the CopyStorm runtime, including the list of Salesforce Objects/Tables to copy. The default view of the Configuration Tab contains core configuration options that are commonly modified. ![](/files/zZQAuW4LNj7rnpz9LVNW) To view additional options, change the “Configuration Level” dropdown: * Standard * Core CopyStorm configuration and performance tuning. * Options used to ensure that CopyStorm does not interfere with other operations (e.g. limiting Salesforce API Call consumption). * Advanced * CopyStorm troubleshooting and logging options. * Developer * CopyStorm application tuning options, advanced network configurations, and query optimization options. ![](/files/Gp7m5o280OFspYhRGVQF) # Core Configuration The Core Configuration Editor is used to specify which objects should be replicated to the database, and to control how long records will be retained in the database after being deleted in Salesforce. ![](/files/yQXoOyf3iGnf6woAk98Y) ## Select Objects The “Select Objects” dropdown selects the objects which will be copied from Salesforce to the database. The easiest way to configure which objects to copy is to select one of the pre-configured options: * All Objects * Copy all Salesforce objects that the CopyStorm Salesforce User has access to. * Automatically copy newly created Salesforce Objects when they become visible to the CopyStorm Salesforce User. * All Objects Except for Files * Copy all Salesforce objects that the CopyStorm Salesforce User has access to, except for Base64/File objects (e.g. Attachment, ContentVersion). * Automatically copy newly created Salesforce Objects when they become visible to the CopyStorm Salesforce User. * All Objects That Can Be Restored * Copy all Salesforce objects that the CopyStorm Salesforce User has access to, except for objects that are known to not be recoverable (e.g. External objects, History objects). * Automatically copy newly created Salesforce Objects when they become visible to the CopyStorm Salesforce User. To configure CopyStorm to copy specific objects, select from the “Manual Selection” options. Selecting a “Manual Selection” will show an “Include” or “Exclude” list. To configure the list, either: * Type or copy-paste the API names of selected objects into the “Select Objects…” section as a comma separated list. * Click on the “Select Objects…” button. * This will cause CopyStorm to connect to Salesforce to determine the Objects available. * A dialog will open allowing selection of available Salesforce Objects. Manual selection options include: * Manual Selection – Include List: Choose Objects to Include * All selected objects will be copied. * Manual Selection – Exclude List: Choose Objects to Exclude * All objects visible to the CopyStorm Salesforce User will be copied, except for selected objects. * Automatically copy newly created Salesforce Objects when they become visible to the CopyStorm Salesforce User. * Manual Selection – Advanced: Choose Objects to Include and Exclude * If the Include list is empty, all Salesforce Objects will be copied, except for objects selected in the Exclude list. * If the Include list is not empty, all selected objects will be copied. ## Delete Older Than The “Delete Older Than” input controls when to remove records that have been deleted from Salesforce from the target database. * 0 (zero): This is the default value. Remove deleted records immediately from the target database. * Blank (no value specified): Mark deleted records as IsDeleted, and never remove deleted records. * A non-zero number: Mark deleted records as IsDeleted, and remove records when they have been deleted from Salesforce for the number of days specified by “Delete Older Than”. CopyStorm discovers deleted records by reading the Salesforce recycle bin. If the recycle bin is emptied before CopyStorm discovers the deletion, or if records have been removed with a hard delete, CopyStorm will not know that the records are deleted. This method is used by some applications. If this is occurring, please use Capstorm’s CopyStorm/Medic tool to discover the deleted records and mark the records as IsDeleted in the target database. ## Include Metadata If the “Include Metadata” checkbox is checked, CopyStorm will include selected metadata components in the backup. By default, if checked, all metadata components are included. Clicking the "Edit" button next to this checkbox opens the metadata selection dialog.
The Metadata Selection Mode dropdown offers three options: * **All Metadata Types** — All available Salesforce metadata types are backed up. This is the default. Any new metadata types added by Salesforce in the future will automatically be included. * **Include Selected Types** — Only the specific metadata types selected from the list will be backed up. Use this to back up a small, known set of types and ignore everything else. * **Exclude Selected Types** — All available metadata types are backed up *except* the ones selected from the list. Use this for broad coverage but need to omit a few specific types. Like "**All Metadata Types**," any new metadata types added by Salesforce in the future will automatically be included. The exclusion list is applied against whatever is currently available, not a fixed snapshot. The dialog also includes: * **Include Managed Packages** — When checked, metadata from managed packages is included in the backup. * **Delete Metadata History after N days** — Controls how long historical versions are retained. Leave blank to keep history indefinitely; set to 0 to keep only the current version with no history. # Salesforce API The Salesforce API Configuration Editor is used to choose which Salesforce API will be used to copy data. ![](/files/40yMhyf6vcl2jWer5nZU) ## Data API Options The Data API Options section is used to configure the primary Salesforce API that will be used to copy data. * Salesforce Threads * The number of Salesforce Objects that will be backed up at the same time by CopyStorm. * The default number is 3, allowable values are 1 – 20. For maximum performance, the optimal value for this parameter is typically 10. * Salesforce API * The default API to use when querying Salesforce. The default API is REST. * Note that CopyStorm will, by default, use REST of any table containing a BASE64 column because this minimizes memory usage for most databases. Common BASE64 tables include Attachment and ContentVersion. ## Bulk API Options The Bulk API parameters section controls when CopyStorm will use the Salesforce Bulk API rather than the default API. Note that: * A Salesforce organization has a limited number of Bulk API calls per 24 hour rolling period. Be aware of your limit. * The SOAP or REST API will almost always perform better than the Bulk API if performance is measured in the amount of time a user has to wait for a backup to complete. * We suspect that this is because Bulk API queries run on a reduced priority thread within the Salesforce infrastructure. | Parameter Name | Default Setting | Required | Description | | ------------------ | --------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bulk Threshold | | Yes |

CopyStorm will only use the Bulk API if the number of records read for a table exceeds the “Bulk Threshold” parameter. If this parameter is left blank then CopyStorm will not use the Bulk API. Examples:

  • If set to 0 then CopyStorm will always use the Bulk API (when possible — not all tables are supported by the Bulk API).
  • If set to 50000 then CopyStorm will switch to the Bulk API after 50,000 records have been processed for a table.

To reserve Bulk API calls for tables where it may help, Capstorm recommends setting this parameter to 50,000 or above.

| | Bulk Limit | | No | Controls the number of records that will be retrieved with each Bulk API call. A larger value will cause each Bulk API call to take longer but minimize the number of Bulk API calls. If this parameter is not specified it defaults to the SOAP Limit for the current table. | | Bulk Polling Rule | | No |

Sets the polling rate CopyStorm uses to check if a Bulk API operation has completed. The format of this parameter is a comma separated list of numbers where each number indicates how many seconds to wait between each poll of Salesforce.

For example, the rule 1,3,5,10,10,10,30 means:

  • Poll Salesforce after 1 second the 1st time.
  • Poll Salesforce again 3 seconds after the 1st check.
  • Poll Salesforce again 5 seconds after the 2nd check.
  • Poll Salesforce again 10 seconds after the 3rd through 5th check.
  • Poll Salesforce every 30 seconds after the 5th check.
| | Max Bulk API Calls | | No | The maximum number of Bulk API calls that will be used in a single CopyStorm session. | ## Metadata API Options The Metadata Data API Options section is used to configure the primary Salesforce API that will be used to copy Metadata. * Metadata Threads * The number of Salesforce Metadata Types that will be backed up at the same time by CopyStorm. * The default number is 5, allowable values are 1 – 10. For maximum performance, the optimal value for this parameter is typically 5. # Guardrails The Guardrails Tab is used to limit the potential impact of individual CopyStorm runtimes and to ensure that CopyStorm jobs do not interfere with other operations. ![](/files/55JEl23QFJ84WQW3LYde) | Parameter Name | Default Setting | Required | Description | | ------------------------------ | --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | API Call Limit | Blank | No | Limits the maximum number of transactions (queries) to send to Salesforce. This option can help avoid exceeding your organizational limit for transactions per 24-hour period. CopyStorm will stop when this value is exceeded. Usually this parameter is only set for the initial load of the CopyStorm database. | | API Threshold | Blank | No | Maximum starting API utilization. When CopyStorm is initializing the data copy, if the current Salesforce API usage is above the provided percentage, the copy job is not started. | | Maximum Runtime | Blank | No | The maximum number of minutes CopyStorm will run before gracefully shutting down. Leave this blank to disable the time limit. | | Production Database Protection | Checked | No | If Checked, CopyStorm will not allow a production Salesforce database to be overwritten by another Salesforce’s data. | | Schema Only Backup | Unchecked | No | If Checked, CopyStorm will back up the Salesforce schema, but Salesforce data will not be included. This option is typically used during Salesforce migrations. | # Maintenance The Maintenance Configuration Editor is used when troubleshooting replication issues. ![](/files/aZ3n0mvQQy7bWT1HLomQ) | Parameter Name | Default Setting | Required | Description | | -------------------------- | --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Modified Since | Blank | No |

If a date is specified, then only records created or modified on or after the specified date are copied. Built in relative dates are available from a pull-down menu or a fixed date can be entered. When entering a date, the interface will have a yellow background which clears when a date is recognized.

Fixed date formats are:

  • dd-MMM-yy (19-May-1984)
  • M/d/y (5/19/84)
  • dd.MM.yy (19.5.1984)
  • dd-MM-yy (19-5-84)
  • yyyyMMdd (19840519)
| | Modified Thru | Blank | No | If a date is specified, only records created on or modified up to or on the specified date are copied. See format notes above. | | Ignore Timestamps | Unchecked | No |

If checked, CopyStorm will retrieve records in ID order rather than timestamp order.

WARNING: unless a CopyStorm job runs to completion without errors, using this option will make subsequent incremental updates fail to fetch the proper records.

| | Analyze UNKNOWN\_EXCEPTION | Unchecked | No | When Salesforce returns an UNKNOWN\_EXCEPTION this generally indicates that a SOQL query failed in a way the Salesforce infrastructure did not expect. When this option is checked and CopyStorm is running in GUI mode, CopyStorm will do a rather expensive analysis of the failed SOQL and attempt to determine a very small SOQL statement which exhibits the problem. The very small SOQL statement is what Salesforce support will require (even demand!). | # Field History The Field History Configuration editor is used to enable and configure CopyStorm field history tracking. ![](/files/jQV3UUDn8wpSMIIm2bfs) CopyStorm field history tracking supports the tracking of nearly all field changes in every Salesforce object. If the “Track Field History” checkbox is selected, the default behavior is to track all changes to all fields. To limit field history tracking or to customize retention periods for field history changes, click on the “Edit” button for advanced Field History configuration. Field history changes are kept in a database table named “CopyForceFieldHistory”. For details see: * [CopyStorm Logging Tables](/frequently-asked-questions/how-does-copystorm-work/does-the-copystorm-database-have-logging-tables) ## Field History Editor Field History is controlled by three major elements: * Whitelist — A Whitelist controls which tables (and optionally fields) participate in field history. An unlimited # of white lists can be defined. * Blacklist — A Blacklist controls which tables DO NOT participate in field history * Retention Days — Retention Days controls how many days CopyStorm should keep field history. This can be control on a per table basis and can range from 1 day to forever. To setup Field History rules there are several decisions to make: * Do I want to control what tables are backed up using a Whitelist or Blacklist approach? * By default, how many days should field history records age before they are automatically deleted by CopyStorm? * Are there any tables whose field history records I want to keep longer (or shorter) than the default? The Field History Editor has several major components: * Global Parameters specify constraints applied to all rules. * The Whitelist/Blacklist editor indicates which tables should (or should not) participate in field history tracking. ![](/files/VGIYeoG1JNa7kpfmQxnS) ### **Global Parameters** There are several global field history parameters. * Include Formula Columns parameter controls where changes to formula fields are recorded as field history changes. The default is false. * The Exclude Audit Fields parameter restricts common Salesforce auditing fields from being records in field history. There are several reasons to exclude an audit field. * A field is a timestamp field that changes on every record change. * A field is Salesforce maintained formula field (example: LastViewedDate) * A field is also recorded as separate column in CopyStorm’s field history table. To change the list of excluded audit fields click on the Edit… link the left of the field label. ![](/files/8wbfnVhK68IOiebNeJOB) ### **Whitelist/Blacklist Editor** The Whitelist/Blacklist part of the editor is used to manage what tables participate in field history tracking. * The Whitelist indicates what tables should participate in field history tracking. If the whitelist is empty this means that all tables except those in the blacklists will be tracked. * The Blacklist indicates what tables should not participate in field history tracking. Of course, in practice either the whitelist or blacklist is used but not both. When the white list is not empty the blacklist is effectively ignored. Before configuration the editor looks like: ![](/files/nIkES0m6JYDgDDet1uLU) Select specific tables and/or table patterns to add to the white list by clicking on the appropriate button. ![](/files/ilmHfrXZWONizPQWW9Gi) ![](/files/Pu4UMGgmyc8u8U5wivWC) If the default field retention rule is too short (or long) for specific tables, an extra white can be added to change the retention rule. The example below changes the retention rule for Account and Contact to 1000 days. ![](/files/9EL5PTOj74FGCbCu60Fr) ## Limit Fields Subject to Tracking It is possible to limit the fields tracked for a specific table by edit the table’s rule. Of course, doing this means that any new fields added to a table will not be tracked by default. The following image illustrates how to limit the fields tracked for the Account table. ![](/files/TU3zpi3IjkOGgK6EkZbo) # Database Reset The Database Reset Configuration Editor is used to delete and rebuild portions of the CopyStorm database. Options within the Database Reset Configuration Editor are only used in specific circumstances and should not be left enabled as part of normal backups.
| Parameter Name | Default Setting | Required | Description | | ---------------------------- | --------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Force Truncate | Unchecked | No | If checked, CopyStorm will truncate all data from each table and start the backup of records from scratch. | | Force Rebuild | Unchecked | No | If checked, CopyStorm will drop each table and rebuild it from scratch. | | Keep Deleted Columns | Unchecked | No | If checked, CopyStorm will retain a column in the target database if it has been dropped in Salesforce. By default CopyStorm will drop a column in the target database if it is dropped in Salesforce. | | Refresh Columns on DDL Error | Unchecked | No | If checked, CopyStorm will rebuild and refresh columns if an unrecoverable DDL exception occurs. CopyStorm will create a new column, requery Salesforce for the column's data, and will rename the new column to replace the old column. This toggle is useful when Salesforce field definitions change in such a way that the corresponding Alter Column SQL commands fail. | | Rebuild on DDL Errors | Unchecked | No | If checked, CopyStorm will drop and rebuild a table if an ALTER statement on the table causes a DDLexception. | # External Configuration The External Configuration Editor is used to provide external folders used by CopyStorm to read configurations and to store temporary files. ![](/files/6O32IIIG4UpfuaeMf48l) | Parameter Name | Default Setting | Required | Description | | ---------------- | --------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Config Directory | Blank | No | By default, CopyStorm will look for configuration file overrides in the directory “config” within the CopyStorm installation directory. If this value is set then CopyStorm will also look in this directory for configuration file overrides. | | Temp Directory | Blank | No | By default, CopyStorm will store temporary files containing Salesforce Files and Attachments in the default Java temporary directory. If this value is set then CopyStorm will instead store temporary files in the provided directory. Note: If large files are being downloaded from Salesforce, this directory can become many GB in size. | # Logging The Logging Configuration Editor is used to configure CopyStorm logs that are written to the database, STDOUT / STDERR, or using the Apache Commons Logging framework. ![](/files/hCtqyoogCIgJlnMmgJL5) | Parameter Name | Default Setting | Required | Description | | --------------------------- | --------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Database Logs in UTC | Unchecked | No | If checked, timestamps recorded in CopyStorm’s job tracking log tables will be in UTC rather than local time. | | Log Tables with No Activity | Unchecked | No | If checked, CopyStorm will log tables with no activity (no records copied, skipped, deleted, or archived) during a job. | | Log Format | Full | No |

This option controls the amount of detail sent to the CopyStorm log file. Options include:

  • Compact: Display a single line of text for each table that has copy activity.
  • Summary: Display a single line for each table processed (even if no records were copied).
  • Full: Display a verbose amount of logging information.
  • Debug: Display everything Full displays, plus additional debugging information.
| | Log Table Sizes | Checked | No | If checked, log the starting and ending table sizes in bytes. | | Use Apache Logging | Unchecked | No | If checked, logging will be performed with the Apache Commons Logging framework. By default, Log4j will be used. | | Debug Mode | Unchecked | No | If checked, CopyStorm will output logging messages to the debug console (normally hidden). | | Silent Mode | Unchecked | No | If checked, when running in batch mode, CopyStorm will only display errors to stderr and will not display debug or informational messages. | | SFDC Logging | Off | No |

This option enables CopyStorm-to-Salesforce job logging. Options include:

  • On: Write job logging information to Salesforce (If Salesforce logging is not supported, logging information will not be pushed and the job will proceed normally).
  • Off: Do not write job logging information to Salesforce.
  • Required: Write job logging information to Salesforce (if Salesforce logging is not supported, the job will abort).

CopyStorm contains backend support only for Salesforce Logging. To enable Salesforce support for this feature, contact your Capstorm account representative.

| | SFDC Logging Interval | 2 | No |

This option sets the time interval (in minutes) between pushes of job logging information to Salesforce.

CopyStorm contains backend support only for Salesforce Logging. To enable Salesforce support for this feature, contact your Capstorm account representative.

| | SFDC Log Count | 4 | No |

This option configures how many job logs for the CopyStorm configuration file are kept in Salesforce. When CopyStorm creates a new job log, older job log records are deleted, retaining records according to this option. For example:

  • With a value of 0, only the most recent job will be retained.
  • With a value of 5, only the most recent job log and the 5 previous job logs will be retained.
  • If this field is left blank/empty, all job logs will be retained.

CopyStorm contains backend support only for Salesforce Logging. To enable Salesforce support for this feature, contact your Capstorm account representative.

| | SFDC Log Retention | 4 | No |

This option determines how long job logs for all configuration file are kept in Salesforce before older log records are deleted. Job logs that are older than this many days are deleted. This option overrides the SFDC Log Count option if the SFDC Log Count option would cause job logs to be retained for more than this many days. For example:

  • With a value of 0, only the most recent job will be retained.
  • With a value of 5, only the most recent job log and logs from the previous 5 days will be retained.
  • If this field is left blank/empty, all job logs will be retained.

CopyStorm contains backend support only for Salesforce Logging. To enable Salesforce support for this feature, contact your Capstorm account representative.

| # Database Networking The Database Networking Configuration Editor contains networking and SQL parameters used when communicating with the CopyStorm database. ![](/files/EfR7BwrWrIzAtyS3a439) | Parameter Name | Default Setting | Required | Description | | ---------------------- | --------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Destination Batch Size | 100 | No | Limits the number of rows that will be written to the target database in a single batch. A smaller number minimizes memory requirements but increases run time. A larger value increases memory requirements and decreases run time. | | DDL Timeout (seconds) | Blank | No | Set the number of seconds CopyStorm will wait for a DDL operation to complete. DDL operations, like CREATE INDEX, may take considerably longer than DML operations and thus have their own timeout parameter. | | # Database Buffers | 10 | No | Sets the number of output buffers to use when running in multi-thread mode. A larger value increases memory requirements but allows Salesforce to read more records before they are written to the target database. | | Rows Per Multi-Insert | Blank | No | If specified then multiple rows will be sent to the database in a single INSERT statement. Please note that not all databases support this feature and when the database is on a local network there is unlikely to be any performance gain by enabling this feature. | | Enable CSV Bulk Import | Checked | No | If set and is supported by the database then CSV bulk imports will be used. For analytical databases like RedShift or Snowflake this option should almost always be selected. | # Database Optimization The Database Query Optimization Parameter Editor controls optimizations used when generating SQL queries for the CopyStorm database. ![](/files/tST3IZ3camkeAKKpHuUI) | Parameter Name | Default Setting | Description | | ------------------------------ | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Optimize Insert SQL | Unchecked | If checked, CopyStorm will not include NULL data in generated INSERT statements. For wide tables with many NULL values this may be considerably faster than unoptimized INSERT statements. | | Optimize Update SQL | Checked | If checked, CopyStorm will compare data from Salesforce with existing data in the CopyStorm database and build minimal SQL UPDATE statements. For wide tables with few changes this may be considerably faster than unoptimized UPDATE statements. | | Optimize Timestamps | Checked | If checked, then use the time of the most recently complete CopyStorm job to reduce the number of records read from Salesforce. If not checked, then the timestamp of the newest record in Salesforce is used to build SOQL queries, and at least one record from each table will always be read. | | Force Updates | Unchecked | If checked, CopyStorm will write all records read from Salesforce to the target database. Records will be updated even if the timestamp is the same as the Salesforce timestamp. This option can be used in conjunction with the “Modified Since” and “Modified Thru” parameters to force CopyStorm to re-read records from Salesforce and update any fields that are out of sync. | | Optimize Deleted Record Scan | Checked | If checked then CopyStorm will use information from the previous backup to minimize the number of records scanned when searching for deleted records. This option is ignored if the Delete Older Than option is blank. | | Update Using DELETE/INSERT | Unchecked | If checked then CopyStorm will UPDATE records by first deleting them and the re-inserting them. This option is ignored if the Optimize Update SQL option is checked. | | Optimize Update Classification | Checked | If checked, CopyStorm will try to avoid database queries used for deciding if records are inserts or updates. If this parameter is unchecked, CopyStorm will always query the database to decide if a record should be inserted or if a pre-existing record should be updated in the database. For example, if a table is empty when CopyStorm starts to copy data, then there is no reason to look for updates - all records will always be inserts. | | Optimize Snapshot Purge | Checked | When checked, Snapshot record cleanup will happen after all records are copied from Salesforce. When unchecked, Snapshot records will be cleaned up after each object is replicated. | # Database Schema The Database Schema Configuration Editor contains parameters used to configure CopyStorm database schema generation. ![](/files/NTlpzPSywsdQb8Vy3Djq) | Parameter Name | Default Setting | Required | Description | | ---------------------- | --------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Use Strict DECIMALs | Unchecked | No |

CopyStorm uses a relaxed definition for all Salesforce numeric columns. This is because it is possible for Salesforce to store values that do not match the declared field definition. An example is the computed column Opportunity.Amount, which can be forced to exceed the field size declared by Salesforce.

By default, CopyStorm uses:

  • NUMERIC for Oracle and PostgreSQL databases.
  • DECIMAL for SQL/Server, MySQL, and H2 databases.
| | Extra Precision | 2 | No | Increases the numeric precision stored by the database by this number of digits, both before and after the decimal point. The default value is almost always a good choice and works around the handful of instances where Salesforce “lies” about the precision of a column. | | AnyType Length | Blank | No | Allocate this many characters for Salesforce columns using the AnyType data type. Use of an AnyType column is rare and only seen in Salesforce system level tables. The default length is 255. | | Use Database Views | Checked | No | If checked, CopyStorm will create database views for Salesforce objects that can be represented as such. For example, all Feed objects can be represented as views on FeedItem. | | Obfuscate Schema | Varies | No | If checked then CopyStorm will obfuscate database table and column names. | | Use NVARCHAR2 | Unchecked | No | If checked and using Oracle, use NVARCHAR2 for String data. If unchecked, assume that the encoding used by Oracle is capable of representing all UTF8 characters. This option is only used when copying to Oracle databases. | | Table Prefix | Blank | No |

If set, all Salesforce backup tables created by CopyStorm will be prefixed with this value.

For example, if “Table Prefix” is set to “SF\_” then the Account table will be created as “SF\_Account” rather than “Account”. Note that this parameter has no effect on tables that have already been created by CopyStorm.

| | Continue on DML Errors | Checked | No |

If checked, CopyStorm will continue processing if a DML error is encountered. A DML error typically occurs when the structure of a Salesforce table has changed in a way that causes an “ALTER TABLE COLUMN” command to fail in the target database. The simplest fix is to drop the table in the target database and let CopyStorm rebuild it.

If checked, CopyStorm will log an exception, skip the problem table, and continue with the next table.

| | Include Formulas | Checked | No | If checked, CopyStorm will create columns corresponding to formula fields in Salesforce and will copy down the formula data into the columns. If unchecked, CopyStorm will ignore formula fields. | | Include Base64 | Checked | No |

This option controls how CopyStorm will handle Base64 fields (file data). Possible values are:

  • Default – CopyStorm will only copy file data if the database supports long CLOB columns.
  • Yes – CopyStorm will always try to copy file data, whether or not the database supports it.
  • No – CopyStorm will never try to copy file data even if the database supports it.
| | Chunk Base64 Values | Unchecked | No | If checked, CopyStorm will store all Base64 values as chunks. These chunks will be specified in the “CopyForceTableFieldChunk” table in the CopyStorm database. | | Synchronize Schema | Checked | No | If checked, CopyStorm will create and alter database columns to match Salesforce definitions/metadata. If unchecked, CopyStorm will not attempt to alter database schema definitions when Salesforce object/field definitions change. | | Auto Upgrade Schema | Checked | No |

Upgrade the database schema to match the schema for this version of CopyStorm.

If checked:

  • The CopyStorm Database will be updated without explicit user approval.

If unchecked:

  • The user will be prompted with planned CopyStorm database additions, deletions, or changes. The database upgrade will not occur until the user approves the action.
  • When running in headless mode, CopyStorm will terminate with an Exception indicating the need for database schema upgrade approval.
| # Database Indexing The Database Indexing Configuration Editor is used to enable or disable automatic database index creation. ![](/files/EkUK2Ig5irxf19qmCjb2) | Parameter Name | Default Setting | Required | Description | | ----------------------- | --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Create Indexes | Checked | No | If checked, CopyStorm will create database indexes for each reference field. Enabling index creation will significantly improve the table join speed in larger databases. Note that an index is always created on each table’s “timestamp” column. | | Create Indexes Early | Checked | No | If checked then CopyStorm will create indexes on Salesforce reference fields when a table is initially created. If not checked then reference field indexes will be created after the first backup of a table. This option can avoid a long delay or timeout when indexes are originally created on tables with many rows. | | Create IsDeleted Index | Checked | No | If checked then CopyStorm will create an index on the fields (IsDeleted, SystemModstamp). | | Rebuild Timestamp Index | Unchecked | No | If checked, the index on the SystemModstamp column will be rebuild if enough new records are inserted. In some database configurations, this can increase performance dramatically. | # Salesforce Network The Salesforce Network Configuration Editor contains networking and SQL parameters used when communicating with Salesforce. ![](/files/gKeuydl4PR1nNCylnChg) | Parameter Name | Default Setting | Required | Description | | -------------------------- | --------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | # Reconnect Attempts | 5 | No | The number of times to try to automatically reconnect with Salesforce if the connection to Salesforce has dropped. | | Reconnect Delay | 15 | No | The number of seconds to wait between attempts to reconnect to Salesforce. | | Backup Rich Text Images | Checked | No | If checked then CopyStorm will replicate images embedded in Salesforce Rich Text fields and Knowledge Articles. Images are stored in the tables CopyForceRtImage and CopyForceImageData. | | Backup URL Images | Checked | No | If checked then CopyStorm will replicate images linked to in URL Image fields. Images are stored in the tables CopyForceURLImage and CopyForceImageData. | | Ignore Numeric Overflow | Checked | No | If checked, then if a numeric formula column contains a value which would cause a numeric overflow in the database then the database value will be set to null. The most common cause of this issue is an ill-defined Salesforce formula column with a near divide by zero error. | | Max Base64 Reader Threads | Blank | No | Set the maximum number of base64 values that will be read concurrently when processing tables containing base64 columns using the REST API. This option can greatly increase the rate at which tables like Attachment and ContentVersion are backed up by downloading the raw file content for multiple records concurrently. | | Continue On Missing Base64 | Blank | No | If checked, then if a Base64 file body cannot be downloaded CopyStorm will report an error and continue replicating additional files. If unchecked, then CopyStorm will halt the replication for a Base64 object when the first unreadable Base64 entry is encountered. | # Salesforce Optimization The SOQL Query Optimization Parameter Editor controls optimizations used when generating SOQL queries for Salesforce. ![](/files/3ByhE8J3SLABr0ekowi8) | Parameter Name | Default Setting | Required | Description | | --------------------------- | --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Estimate Total Time | Unchecked | No | If checked and CopyStorm is running with a GUI, an extra SELECT query will be issued for each table to determine the total number of records to be copied. | | SFDC Commit Latency | Blank | No | The number of seconds a Salesforce records has to be unchanged until it can be trusted that records have been written to Salesforce in timestamp order. This is a workaround for Salesforce’s belief in time travel — a record with a timestamp of 12 noon can be written to Salesforce before one with a timestamp of 11:59 am. | | Timestamp Latency | 900 | No | The number of seconds to re-query based on the maximum timestamp from the previous CopyStorm job. This parameter is used to capture record changes that may occur out-of-order based on Salesforce record save behavior – i.e. changes that happen after SystemModstamp values are assigned, but before the record is committed. For more details on Salesforce’s record save behavior see [this article](https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_triggers_order_of_execution.htm). | | Row LIMIT per Query | Blank | No | The default number of records to read from Salesforce with a single SOQL statement. A larger number may reduce the number of Salesforce API calls slightly but also may result in query time outs that CopyStorm must recover from using additional API calls. The default value is 20000. | | Always Validate SOQL | Unchecked | No | If checked then CopyStorm will validate every generated SOQL statement to insure that a QUERY\_TOO\_COMPLEX error is not generated. Since CopyStorm dynamically recognizes QUERY\_TOO\_COMPLEX errors and rewrites queries to avoid them there is little need for this parameter to be set unless the user is a CAPSTORM developer. | | Find Optimal SOQL | Unchecked | No | If check then CopyStorm will take extra effort to generate to optimal load balanced SOQL statements when multiple SOQL statement are needed to read a single row from a table. This parameter is for CAPSTORM developers. | | Max Simple Query | Blank | No | The smallest SOQL query that CopyStorm will assume never will generate a QUERY\_TOO\_COMPLEX error. This parameter is for CAPSTORM developers. | | Enable SOQL Query Lookahead | Checked | No | When checked, CopyStorm will read the next batch of record for a table while processing the current batch. This improves the overall performance of CopyStorm with a slightly larger load on the local computer. | | Validate SOQL End of Data | Checked | No | If checked then CopyStorm will verify that the backup of a table is complete by running a single validation query. This is necessary in cases where SOQL returns no records from a query when there are records available. | # Snapshot Tab The CopyStorm Snapshot tab is the launch point for creating, editing, using, and exploring CopyStorm snapshot data sources. A CopyStorm Snapshot is a database store containing tamper-proof snapshots of records from any Salesforce table where the snap-shots are taken based upon rules specified by a user. CopyStorm Snapshots can be used to store many types of data. * A Snapshot can be used to create a unlimited duration recycle bin or a recycle bin where data retention policies vary by table. * A Snapshot can be used to permanently save a copy of Salesforce records on demand. * A Snapshot can be used to track all versions of critical Salesforce records. Each CopyStorm Snapshot has policies which determine which table and which records should be snapshot. Like the core CopyStorm backup process, Snapshot policies can be applied automatically as a regular part of a backup or can be applied on demand. ## The CopyStorm Snapshot GUI The CopyStorm Snapshot tab lists all currently defined snapshots, can be used to create a new snapshot, and is the launch point for most Snapshot operations including: * [Edit a Snapshot Policy’s Rules](/reference/archive-tab/archive-policy-editor) * [Explore the data in a Snapshot](/reference/archive-tab/explore-archive) * [Delete a Snapshot Policy and its data](/reference/archive-tab/delete-archive-policy) * [Run a Snapshot Policy on demand.](/reference/archive-tab/run-archive-policy) * [Apply a Snapshot Policy’s data retention rules.](/reference/archive-tab/run-archive-policy-retention-rules) ![](/files/y5j5OTifWthygQ9k1lXC) ## Next Steps To get started with CopyStorm Snapshots, please review the following: * [CopyStorm Snapshot System Overview](/frequently-asked-questions/how-do-copystorm-archives-work/copystorm-archival-system-overview) Once you have the basics down try one of these tutorials: * [How to Setup the Capstorm Recycle Bin](/frequently-asked-questions/how-do-copystorm-archives-work/how-to-setup-the-capstorm-recycle-bin) * [How to Setup a Custom Snapshot Policy](/frequently-asked-questions/how-do-copystorm-archives-work/how-to-setup-a-custom-archive) * [How to Explore and Export Snapshot Data](/frequently-asked-questions/how-do-copystorm-archives-work/how-to-export-data-from-an-archive) Finally, jump to any of the following reference articles: * [Snapshot Policy Editor](/reference/archive-tab/archive-policy-editor) * [Refresh Snapshot Statistics](/reference/archive-tab/refresh-archive-statistics) * [Run a Snapshot Policy](/reference/archive-tab/run-archive-policy) * [Delete a Snapshot Policy](/reference/archive-tab/delete-archive-policy) * [Snapshot Explorer](/reference/archive-tab/explore-archive) # Snapshot Policy Editor CopyStorm Snapshots support an unlimited number of Snapshot Policies with each Policy defined by the *Snapshot Policy Editor.* The following image lays out the primary sections of the *Snapshot Policy Editor*. ![](/files/yBSVDAVDDNV7wN0W8tFx) There are several tabs on the Snapshot Policy Editor. * The *Main* tab contains properties common to all snapshots and is where the *Filter Rule Type* for the snapshot is indicated. * The *Filter Rule Parameters* tab contains filter specific data. For example: * Most filter types support selecting which tables to include in a snapshot. * The *Column Expression* filter type supports rich user supplied expressions to limit which records are snapshot. * The *Advanced* tab contains parameters rarely modified by a customer. The following subsections describe each tab in details. ## Main Tab ### **Name** Each Snapshot Policy requires a unique name. There are no restrictions on what characters can appear in a name but the name itself must be unique for each snapshot policy. ### **Active** A Snapshot Policy is always in either an *Active* or *Inactive* state. * A policy in an *Active* state will be be used by CopyStorm as part of a normal backup. For example, the *Default Recycle Bin* policy will usually be set to an *Active* state so the backup will automatically snapshot records as they are deleted from Salesforce. * A policy in an *Inactive* state can be ran by hand but will not be automatically processed as part of a CopyStorm backup. *Inactive* policies are useful for one-off compliance needs and snapshot policies that do not need to be part of a scheduled process. ### **Apply Policy** The *Apply Policy* section controls when CopyStorm will check a record in the database against the policy filter. There are three options: * After Insert — check a record against the policy immediately after it is inserted into the CopyStorm database. * Before Update — check a record against the policy immediately before it is updated with changes from Salesforce. * After Update — check a record against the policy immediately after it has been updated in the CopyStorm database. One or all of the option can be selected concurrently. In all cases, CopyStorm will not add a record to a snapshot if an identical copy of the record is already in the snapshot. ### **Also Snapshot** To reduce the amount of database space required to snapshot a record, certain non-essential fields can be excluded. Common exclusion choices are: * Empty Fields — when empty fields are excluded a savings of the “length of the field name” plus 3 bytes is saved for each of the record’s empty fields. * Formula Fields — excluding a formula field can often save a lot of space in a record. Formulas are computed, so there is no data loss. ### **Days to Retain** The *# Days to Retain* parameter controls how many days records are kept in a snapshot before they are deleted. * Blank means to keep snapshot records forever. * 90 means to delete snapshot records after 90 days. In addition to the global # Days to Retain parameter, the policy for individual tables in a snapshot can be set explicitly by creating an override rule using the *Advanced..* button. Clicking on the *Advanced…* button launches the [Advanced Retention Rules Editor](/reference/archive-tab/archive-policy-editor/advanced-retention-rules). ### **Filter Rule Type** The Filter Rule Type specifies specific properties that a record must have to be considered by the snapshot policy. Example Filter Rule Types include: * Deleted Records — a record passes the filter if it has been deleted in Salesforce. * Column Expression — a record passes the filter if the data in its columns matches a user defined expression. * All Changes — a record always passes the filter. For further details for each filter rule type, see the article on [Policy Filter Rules](/reference/archive-tab/archive-policy-editor/policy-filter-rules). ## Filter Rule Parameters Tab The content of the *Filter Rule Parameters* tab depends on the *Filter Rule Type* selected on the *Main* tab. For example, the following screen shot is for a *Column Expression* filter rule. * This filter says to only snapshot a record if it is a Contact or Account and is named CapStorm. * Like most filters, the tables to be processed by the rule can be specified as a White List, Black List, or both. For further details for each filter rule type, see the article on [Policy Filter Rules](/reference/archive-tab/archive-policy-editor/policy-filter-rules). ![](/files/HduCA99WckIGUEDpalWc) ## Advanced Tab To boost performance of record relationship traversal, CopyStorm Snapshots maintain a high speed database index. This section supports excluding reference fields from the high speed index to save space in the database. The reference fields selected by default are rarely needed for relationship traversal, occur on nearly every record, and thus can take up significant space in the CopyStorm Snapshot index. ![](/files/KjZ3tRFrQiGB1DHXzZQ2) # Advanced Retention Rules Though every Snapshot Policy has a default record retention rule, it is often convenient to have a custom retention rule for specific tables. Example: * By default, retain records in a snapshot for 90 days. * Retain Account, Contact, and Opportunity records for 365 days. * Retain Case records for 2 years. * Do not retain EmailMessage records. The following two screens illustrate how to set up this policy. ![](/files/BlG2qpdAUuU4PI4vtKdj) Clicking on the *Advanced* button launches an editor to override the default record retention rule. The following example sets up the scenario described at the top of this page. ![](/files/YTXVDSeJdUHE0PDgDEQX) # Policy Filter Rules A Snapshot Policy Filter Rule contains the logic which determines if a database record should be processed by the snapshot policy. There are several types of policy filters built into CopyStorm Snapshot. * [All Changes](/reference/archive-tab/archive-policy-editor/policy-filter-rules/all-changes-filter-rule) — this filter always indicates that a record should be processed. Use it when all changes to records should be snapshot. * [Column Expression](/reference/archive-tab/archive-policy-editor/policy-filter-rules/column-expression-filter-rule) — with this filter a rich expression based on columns in a record can be used to determine if a record should be snapshot. * [Deleted Records](/reference/archive-tab/archive-policy-editor/policy-filter-rules/deleted-records-filter-rule) — this filter only matches records that have been deleted in Salesforce. This filter is used to build recycle bin type policies. * [SQL Predicate](/reference/archive-tab/archive-policy-editor/policy-filter-rules/sql-predicate-filter-rule) — this advanced filter supports injecting user specific SQL fragments into a query that determines if a record should be snapshot. The Filter Rule for a Snapshot Policy is specified on the *Main Tab* of the snapshot editor. Parameters specific to the selected *Filter Rule Type* can be set on the *Filter Rule Parameters* tab. ![](/files/KfD1B3MR8txwS0tL4sU5) ## White and Black Lists Nearly all Filter Rule Types support the concept of a White and/or Black list for selecting the tables to be processed by the policy. * A White List indicates tables to include in the snapshot. * A Black List indicates tables to exclude from the snapshot. All tables not on the black list will be included. Typically the technique used for selecting tables for a snapshot is either by White List or Black List. However, if you are brave (and using regular expressions) both a White and Black list may be useful. The following example (from the *All Changes* filter rule type) illustrate the components of the White/Black list editor. * The pull-down menu indicates whether tables should be selected by White list, Black list, or both. ![](/files/JDzzllsO7Ppa11XiE1dJ) The dialog for selecting tables is nearly the same for a White or Black list. The following example is for a White list. ![](/files/4Azi92qImZUJiAjQCFJi) # All Changes Filter Rule The *All Changes Filter Rul* e always indicates that a record should be snapshot for a change to any field. Basically, if a caller asks if a record should be snapshot then the *All Changes Filter Rule* will always respond with “Yes.” This rule is useful for any table where all changes should be included in the snapshot. Since the *All Changes Filter Rule* always says “Yes, this record should be snapshot”, naturally its only configuration parameter is [which tables to include in the snapshot](/reference/archive-tab/archive-policy-editor/policy-filter-rules#white-and-black-lists). ![](/files/MWeNWnZYVUhpRNF3Mu7R) # Column Expression Filter Rule A Column Expression Filter Rule is a rich graphical tool for specifying the conditions under which a record should be added to a CopyStorm snapshot. The interface is patterned after the Salesforce Report Writer expression builder and should feel familiar to most users. The first step, after selecting the Column Expression option from the Filter Rule Type pull-down, is to launch the column expression filter editor. ![](/files/PzQMug1tdOMRj4EPAY3j) Clicking on the *Edit Column Expression Filter* button will launch the editor. ![](/files/Mvk0Yfj0KGHagZhx8omL) The following example creates a filter that finds records where: * the Name field contains “Louis” The screen image shows the dialog after the *Refresh Preview* button has been clicked. Notice that the searched database has multiple accounts with the similar names that matched the filter. ![](/files/yHJOxCl6lWJEBU3bL0d2) After saving changes and returning to the policy editor, a summary of the filter will be shown. To change the filter simply click on the *Edit Column Expression Rule* again. ![](/files/uNNsbYA1DHMiwKyn1am4) ## Common Issues ### Issue 1: Very few columns are available for selection The list of available columns is determined as the intersection of all columns from table names matched by the policy’s *Include Tables* list. If you see a single column named *Id* then the policy is including all tables in Salesforce and the only common column name is *Id*. To increase the number of available columns return to the policy editor page and adjust the list of included tables. ### Issue 2: No Records appear when I click the *Refresh Preview* button. This issue only occurs when there are no matching records in the CopyStorm database in any of the selected tables. The solution is to modify the query. # Deleted Records Filter Rule The *Deleted Records Filter Rule* indicates that a CopyStorm record should be snapshot only when its IsDeleted flag is set to true. The primary use of the filter is to manage the records added to a recycle bin. Though CopyStorm is initialized with a policy, *Default Recycle Bin*, that uses this filter rule, a user can added additional snapshot policies based on a *Deleted Records Filter Rule* if a non-standard recycle bin is desired. The only configuration for a *Deleted Records Filter Rule* is [which tables to include in the snapshot](/reference/archive-tab/archive-policy-editor/policy-filter-rules#white-and-black-lists). ![](/files/sQ52jPHBWRN73X7ad9BW) # SQL Predicate Filter Rule A *SQL Predicate Filter Rul* e is useful when other filtering techniques are insufficient and it is possible to write a database SQL fragment to identify records that match a snapshot policy. It is rare to need this filter rule because filtering can typically be accomplished using the graphical *Column Expression Filter Rule*. A *SQL Predicate Filter Rule* has a single configuration parameter: a fragment of database specific SQL will be added to the WHERE clause of a SQL statement generated by CopyStorm. Example: * `SELECT Id FROM Account WHERE Id IN (…) AND (configuration parameter from SQL Predicate Filter)` If the argument to a *SQL Predicate Filter Rule* was “RecordAge\_\_c>46 OR alwaysSnapshot\_\_c=1” then SQL statements generated by CopyStorm would look like: * `SELECT Id FROM Account WHERE Id IN (…) AND (RecordAge__c>46 OR alwaysSnapshot__c=1)` A *SQL Predicate Filter Rul*e also includes a rule for [which tables to include in the snapshot](/reference/archive-tab/archive-policy-editor/policy-filter-rules#white-and-black-lists). ![](/files/ONdA9lKxqbQCSsI7oCpo) # Explore Snapshot The *Snapshot Explorer* is a tool for viewing and exporting data from a CopyStorm Snapshot. The tool is launched from the CopyStorm Snapshot tab. ![](/files/vvPr14YvePnDNy698zn7) The Snapshot Explorer has four stages, indicated by the navigation bar at the top of the dialog. * Start — Select the snapshot to explore. * Select Table — Select a table from the Snapshot to explore. * Preview — Build a query and preview the selected data from the Snapshot. * Export — Export the data from the snapshot in a variety of formats. ## Start Stage ![](/files/G3EA0fb96dWYmu9njyGb) ## Select Table Stage In the Select Table stage, select the table you wish to explore and optionally select the columns in the table to view. Note that you can always return to this stage in order to update the selected columns. ![](/files/I8iWqZnznqWWkjaPON1b) ## Preview Stage In the *Preview Stage*, optionally create a filter that limits the records to view or export. The filter is created by dragging and dropping elements from the left hand side of the screen to the big white box on the right. In the example screen shot a filter for finding records with a *capstorm* email domain has been specified and then the *Refresh Preview* button was clicked. ![](/files/EcEyCTsoDqWctnRWOYpF) ## Export Stage In the Export Stage: * Review and update the list of columns to export. * Click on the *Start Snapshot Export* button and the system will display a typical *Save As* dialog. By default, only the most recent version of a record will be written in the export. To override this behavior and write ALL VERSIONS of a record to the export check the box to *Include All Snapshot Record Versions of a Salesforce Record*. Of course, even if this option is checked, a record still has to pass the filter criteria from the *Preview* stage. ![](/files/8k1ek2lKGsvKDXWDHRkK) The Save As dialog which appears after clicking on Start Snapshot Export will look like your system’s default dialog with the addition of a file format selection section. ![](/files/dpZV86DKq303Huj3QYha) ![](/files/vWrSNMSDmBK9VKc1LQih) When the result of an export is loaded into your favorite spreadsheet it will look like: ![](/files/AOFTeLKCPkXIbKJJfYLV) # Delete Snapshot Policy Though it is possible to delete a Snapshot Policy, usually a site will simply want to make the policy inactive. When a Snapshot Policy is deleted both the policy definition and all data snapshot by the policy are deleted from the CopyStorm database. The Snapshot policy deletion tool is launched using right-click context menu on the CopyStorm Snapshot tab. ![](/files/F281ixRLAaEad1ZQT6Bu) ![](/files/VHUcEyyF1tCs9TnKGeH4) # Run Snapshot Policy The Run Snapshot Policy feature is used to run a snapshot policy against existing data in a CopyStorm snapshot on demand. This process is useful in a number of circumstances. * Suppose a new snapshot for all Cases older than 2 years was just defined. Before activating the snapshot policy (so it will run automatically when a backup happens) you may want to snapshot all existing cases in the CopyStorm database that are older than 2 years. * Your compliance department has told you to snapshot all records related to ACME, Inc. because they have filed a breach lawsuit against the company. In this case the snapshot policy that you define may run just once — on demand — and never be scheduled to run automatically. The Run Snapshot Policy feature is accessed from the right-click context menu on the CopyStorm Snapshot tab. ![](/files/APcIjS9SUJKNngB0ziwP) There are just a few options to run a Snapshot Policy by hand. * *Scan Records From* sets a start date from which records will be searched. By default the start date will be the date of the earliest record in the corresponding CopyStorm table. * *Scan Records Thru* sets the last date on which records will be searched. By default the end data will be the most recent date in the CopyStorm database. * *# Concurrent Searches* limits the number of tables that will be processed concurrently. You can set it to a large value (like 200) but be aware that a large number will consume considerable database resources. * *# Searches Per Batch* limits the number of records that will be evaluated by the snapshot policy concurrently (on a per table basis). A larger number will increase performance slightly at the cost of making the user interface a bit less responsive. * *# Snapshot Per Batch* limits the number of records that will be written to a snapshot in a single database transaction. ![](/files/kljsxblQinjun3kmwqxW) While a Snapshot Policy is running the screen will continuously update with progress information. At any point in time you can cancel the snapshot process and only records snapshot so far will be preserved in the database. The following screenshot illustrates how the dialog will appear after a snapshot policy has run to completion. Note that are several reasons that a record may be processed but not snapshot. * If a record fails to match the filter associated with the snapshot policy then it will be skipped. * If a record matches the filter associated with the snapshot policy but is already in the snapshot with exactly the same data then it will be skipped. ![](/files/nwlgdBXvs1HOmMEo4XPK) # Run Snapshot Policy Retention Rules The *Run Snapshot Policy Retention Rules* feature is used to force a snapshot to look for records that should be deleted from the Snapshot based on its retention rules and then delete the records. This happens automatically for active *Snapshot Policies* as part of the regular CopyStorm backup process, so this feature is most useful for inactive *Snapshot Policies*. The *Run Snapshot Policy Retention Rules* feature is accessed from the right-click context menu on the CopyStorm Snapshot tab. ![](/files/ZtoTgLiepuQBzYqZCvCm) There are only two parameter options when running retention rules. * Search Batch Size — number of records that will be searched at once for potential deletion. * Deletion Batch Size — number of records that will be deleted in a single database transaction. ![](/files/WvAZFNGq4ChI15j7j49W) # Refresh Snapshot Statistics Like most database systems, the CopyStorm Snapshot system will operate most efficiently when its has statistics about what is stored in snapshots. Example: * If CopyStorm knows what tables have been snapshot by a policy then it can limit searches to just these tables. * If CopyStorm knows how may records are in a snapshot it can generated more efficient SQL queries. For the most part, CopyStorm Snapshot updates statistics without any user intervention. This page, however, describes how to force CopyStorm Snapshot to update statistics if the most accurate statistics possible are desired or if the statistics have been corrupted. Note: it is unlikely that you will ever need this feature. The *Update Statistic* s feature is accessed from the right-click context menu on the CopyStorm Snapshot tab. ![](/files/VTf1wcwm8qNbsHC3aYxx) In the Refresh Statistics dialog there is the option to refresh statistics for a single snapshot or for all snapshots in the system. The following image illustrates what the screen may look like after refreshing statistics for a single snapshot. ![](/files/HLrFtkwgtsYrqO32FCvB) # Run Snapshot Policy from Command Line This page provides details into the command line options for running snapshot policies for CopyStorm in headless mode. This feature was introduced as part of the 10.65.1 GA release of CopyStorm. For more information regarding running snapshot policies, see [here](/reference/archive-tab/run-archive-policy). #### Overview The command line functionality mirrors the same options and functionality as seen in the CopyStorm application using the "Run Snapshot Rules Now" option including: * Start and end date ranges for records to snapshot * Number of concurrent searches (number of objects snapshot in parallel) * Search batch size * Snapshot per batch #### Command Line Switches
Command Line SwitchExplanationRequiredDefault
snapshotThe name of the snapshot to runYesN/A
snapshotThreadsThe number of objects to snapshot in parallelNo10
snapshotSearchBatchSizeThe read batch size from the database for candidate recordsNo1000
snapshotWriteBatchSizeThe write batch size to the database for found candidate recordsNo1000
snapshotScanRecordsFromThe date to start scanning records from (i.e. 2020-01-01)NoEmpty
snapshotScanRecordsThruThe date to end scanning records from (i.e. 2025-01-01)NoEmpty
#### Example Usage This example is for Windows installations of CopyStorm. For Linux/Mac installations, change *CopyStorm.bat* to *CopyStorm.sh* ``` ./CopyStorm.bat -snapshot "Default Point In Time" "/path/to/config.copyStorm" -snapshotThreads 5 -snapshotSearchBatchSize 5000 -snapshotWriteBatchSize 5000 -snapshotScanRecordsFrom "2020-01-01" -snapshotScanRecordsThru "2025-01-01" ``` # Batch File Tab The Batch File tab is used to generate a script that can be used to launch CopyStorm as a batch job. Generated scripts encapsulate everything needed to run a CopyStorm job, including: * The location of CopyStorm. * The location of the configuration file defining backup parameters. For many sites a generated batch file is not necessary due to built-in command line options in the main CopyStorm launch script allowing batch mode execution. Example command lines include: * sh CopyStorm.sh -run /path/to/my/configFile.copyStorm * CopyStorm.bat -run C:\path\to\my\configFile.copyStorm ## The Core Tool The core function of the Batch File tab is to generate a script file that will: * Run CopyStorm using a specific configuration file from any directory using just the name of the script file. * Capture any exception information from CopyStorm to “do something” with the data. By default the script emails exception information to a user but this can easily be adapted to other purposes. ![](/files/yNjFivS2S3vqVZCkQIik) If you have configured the [Notifications tab](/reference/notifications-tab), the process for generating a script file is simple. This is the approach used nearly 100% of the time: * Use the “Script Format” menu to specify the desired scripting language — bash, bat, python, or vbs. * Click on the “Generate Script” button. * Click on the “Save Script” button and save the script to a file. If your site cannot use the built-in Notification system the process for generating a script file is: * Use the “Script Format” menu to select one of the script formats marked as Advanced. * Enter an email address for the notification sent each time the script runs. * Enter an email address for the notification sent if CopyStorm reports an exception. * Enter the name of your local SMTP server. * In the example a public server hosted by Google is used. * Enter the email address that will send the email. * Click on the “Generate Script” button. * Click on the “Save Script” button and save the script to a file. * Edit the generated script’s code to implement your custom notification requirements. After these steps are completed, you own the generated script file and can change it in any way you prefer. If you have trouble getting the email portion to work properly, please read [this FAQ](/frequently-asked-questions/installation/why-is-my-batch-file-not-sending-email). # Notifications Tab The Notifications Tab contains parameters used to control who receives a status email when a CopyStorm job runs and under what circumstances email is sent. CopyStorm supports email notifications via SMTP or via a Salesforce SOAP API call. ## SFDC Notifications ![](/files/aDAcNYVWuhsHPXLALrpE) | Parameter Name | Required | Description | | --------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Send Email To | Yes | Notification emails will be sent to this white space separated list of email addresses. | | Subject | Yes |

The subject line used for notification emails. Though it can be any text you want, there are a few special text values that will be replaced dynamically before a message is sent:

  • “{status}” will be replaced by “OK” or “ERROR” based on the outcome of the job.

    • If there are any exceptions encountered then “ERROR” will be substituted, otherwise “OK” will be used.

  • “{file}” will be replaced by the base name of the CopyStorm configuration file used by the job.

    • For example, the configuration file “C:\capstorm\config\myCopyStormConfig.copyStorm” will be replaced with “myCopyStormConfig.copyStorm”.
| | Send Notification On | No |

Determines when CopyStorm will send a notification email:

  • Success: If checked, an email will be sent if a job works perfectly (no exceptions).
  • Exception(s): If checked, an email will be sent if a job has any exception (including non-fatal exceptions where CopyStorm recovered).
| | Notify When Using GUI | No | If checked, email notifications will be sent when CopyStorm is launched from the GUI. If not checked, email notifications will only be sent when CopyStorm runs in batch mode. | | Notify Current User | No | If checked, the Salesforce User specified on the Main tab of CopyStorm will be notified in addition to the specified recipients (Send Email To). | ## SMTP Notifications ![](/files/2xJUV0qX8HUrIk9cyuT8) | Parameter Name | Required | Description | | --------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Email Host | Yes | The SMTP host used by your organization (used to send email). If you do not know this information ask your email administrator. | | Email From | Yes | The email address that will appear in the “From” field in sent notifications. Note that many email servers allow this value to be a fake email address (e.g. “”). | | Send Email To | Yes | Notification emails will be sent to this white space separated list of email addresses. | | Subject | Yes |

The subject line used for notification emails. Though it can be any text you want, there are a few special text values that will be replaced dynamically before a message is sent:

  • “{status}” will be replaced by “OK” or “ERROR” based on the outcome of the job.

    • If there are any exceptions encountered then “ERROR” will be substituted, otherwise “OK” will be used.

  • “{file}” will be replaced by the base name of the CopyStorm configuration file used by the job.

    • For example, the configuration file “C:\capstorm\config\myCopyStormConfig.copyStorm” will be replaced with “myCopyStormConfig.copyStorm”.
| | Send Notification On | No |

Determines when CopyStorm will send a notification email:

  • Success: If checked, an email will be sent if a job works perfectly (no exceptions).
  • Exception(s): If checked, an email will be sent if a job has any exception (including non-fatal exceptions where CopyStorm recovered).
| | Notify When Using GUI | No | If checked, email notifications will be sent when CopyStorm is launched from the GUI. If not checked, email notifications will only be sent when CopyStorm runs in batch mode. | | Require Encryption | No | If checked, CopyStorm will require that connections to the email server be encrypted with TLS. | | Email Report Format | Yes | A list of supported formats for the notification email. | | Email User | No | Username used to authenticate with the SMTP server. | | Email Password | No | Password used to authenticate with the SMTP server. | ## How to Test Email Delivery The “Test Email Delivery” button causes an email to be sent to the recipient list. If communication to the SMTP email server or target Salesforce org is successful an acknowledgement dialog will be displayed and an email will be delivered to the recipients. If communication to the SMTP email server or target Salesforce org fails, a dialog will be displayed showing the error. In this case you should show the error to your local administrator. Test email notification: ![](/files/0H35wrwXXXRr5rWfKZrS) ## CopyStorm Notification Reports Under normal circumstances CopyStorm will deliver a notification email that looks like the following. Note that this data is also stored in the CopyStorm database and can be browsed using CopyStorm/Medic (or any other tool you choose). ![](/files/nCppEW8GZde4wr5JPveU) If CopyStorm encountered an error during the backup job, errors will be listed below this summary table: ![](/files/sz7XOEhFMSkFERSsD0NX) If CopyStorm fails before any backup can be attempted, a notification email like the following will be sent. In this example the password for Salesforce was incorrect in the CopyStorm configuration file. ![](/files/yy3JepaV2Slgz3lVSsGW) # History Tab Each time CopyStorm runs it records job information and statistics into a set of history logging tables within the CopyStorm database. The History Tab supports viewing and exporting CopyStorm job history. The History Tab consists of two sections — a top section used to search for past CopyStorm jobs and a bottom section displaying a one-line status for each job in the database. ## CopyStorm Job History ![](/files/k9koEAiAMGVLt8CVzwM8) ## Job History Table Headers | Column Name | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | View\... | Opens a dialog providing detailed status of the selected job. | | Id | ID of the job. | | Status | Completion status of the job. Green checkmark indicates this job had no exceptions. Red exclamation indicates job finished with at least one exception. | | Start | The start timestamp of the job, in local time. | | End | The end timestamp of the job, in local time. | | #Copied | Number of records written to the database. | | #Skipped | Records read but not copied as they are already in the backup. | | #Deleted | Records removed from the database due to deletion in Salesforce beyond retention settings. | | #Snapshot | Records captured as part of point-in-time copy or custom snapshot. | | #Exceptions | Number of exceptions reported during the job. | | Time | Total elapsed time of the job. | | API Calls | Number of Salesforce API calls used during the job. | | Version | Application version used for the job. | | Config File | Name of the `.copyStorm` configuration file associated with the job. | ## Individual Job View When the “View…” column is clicked a dialog appears providing detailed job status. This dialog is similar to the “Copy in Progress” GUI but only contains tables that had activity in the selected job. ![](/files/x4peU56NeuXFQmlL969a) ### Individual Job View Table Headers | Column Name | Description | | ----------- | ------------------------------------------------------------------------------------------ | | Table | Opens a dialog providing detailed status of the selected job. | | Start Time | Starting time of the table copy. | | End Time | Ending time of the table copy. | | #Read | Number of records read from Salesforce. | | #Copied | Number of records written to the database. | | #Skipped | Records read but not copied as they are already in the backup. | | #Deleted | Records removed from the database due to deletion in Salesforce beyond retention settings. | | #Per Minute | Average number of records written to the database per minute. | | Total Time | Total Elapsed time of the table. | ### Exceptions View Any exceptions that occurred during the job will be displayed in the Exceptions tab. In this example, the job had a single exception. ![](/files/ZRlNEFUICenSP8HisI3q) ## CopyStorm/Medic Job History ![](/files/pB8nyPMEc0t6nj4EJzgd) ## Job History Table Headers | Column Name | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | View\... | Opens a dialog providing detailed status of the selected job. | | Status | Completion status of the job. Green checkmark indicates this job had no exceptions. Red exclamation indicates job finished with at least one exception. | | Start | Start timestamp of the job. | | End | Records read but not copied as they are already in the backup. | | #Deleted | Records removed from the database due to deletion in Salesforce beyond retention settings. | | #Snapshot | Records captured as part of point-in-time copy or custom snapshot. | | #Exceptions | Number of exceptions reported during the job. | | Time | Total elapsed time of the job. | | API Calls | Number of Salesforce API calls used during the job. | | Version | Application version used for the job. | | Config File | Name of the `.copyStormMedic` configuration file associated with the job. | ## Individual Job View When the “View…” column is clicked a dialog appears providing detailed job status: ![](/files/Wbm0Waiyx0L7DTwfiyBc) ## Other Features In addition to viewing historical job data, the History Tab can be used to delete job records and to export job data to Excel-compatible CSV files. To delete records: * Click on the check box next to one or more Job #. * Click on the “Delete” button. To export job history to CSV files: * Click on the check box next to one or more Job #. * Click on the “Export” button. # Configuration Files CopyStorm supports saving a backup configuration to a file in two formats: * Plain-text XML. * Encrypted text (256-bit DES encryption). Both formats support the optional saving of Salesforce and database credentials. A CopyStorm backup configuration can be completely restored by going to File/Open in the application menu. To save a configuration, go to File/Save in the application menu: ![](/files/1npzEUsPa65ap3M0uucb) The plain-text XML version of the file is fairly complex and not intended to be edited by hand, but it is possible. However, Capstorm may not provide support for problems arising from a hand-edited configuration file. # How to Override Salesforce Field Definitions Copystorm creates database columns using metadata provided by Salesforce to determine the proper database column type. Salesforce metadata is usually correct, but in the rare occasion that it is wrong or inconsistent CopyStorm provides a way to override the column type. This article describes how to configure CopyStorm to override the column definition provided by Salesforce (or even ignore a Salesforce column completely). ## Background * If Salesforce says a column is 80 characters long, CopyStorm will create a NVARCHAR(80) column in the target database. * If Salesforce says a column is a single digit number, CopyStorm will create a DECIMAL(1,0) column in the target database. Unfortunately, sometimes data provided by Salesforce does not match the column definition. For example, Salesforce claims that multipicklist columns may contain up to 4099 UTF-8 characters. Though this is true in fact, it would be rare for a Salesforce instance to have a value this long. For searching, indexing, and data storage purposes a CopyStorm instance may override Salesforce’s 4099 value with a much smaller data type. Sometimes Salesforce metadata is wrong when determining column size — for example, an aggregate column that says it is 80 characters wide may have much wider values when table data is queried (CopyStorm deals with this case automatically). In all cases, Salesforce metadata can be overridden by including an *ExtractionManagerConfig.xml* configuration file. The configuration file can be located in: * The configuration directory referenced by the .copyStorm configuration file. * The directory “config” in the CopyStorm installation directory. It is also possible to force CopyStorm to completely ignore a field using directives in an *ExtractionManagerConfig.xml* file. See the end of this article for details. ## FieldOverride Options The only tricky part of this configuration is that table and column names must precisely match the definitions in Salesforce (names are case sensitive). There is no limit to the number of FieldOverride directives. New and changed directives will be applied to existing columns in the target database the next time CopyStorm runs. | Option Name | Value Type | Description | | -------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------- | | table | string | Salesforce table to which the override applies. This option is case sensitive. | | column | string | Salesforce column to which the override applies. This option is case sensitive. | | length | integer | Number of characters for a string column. This option only applies to string type columns. | | digits | integer | Total number of digits for a numeric column (utilized for integers). | | precision | integer | Total number of digits for a numeric column (utilized for decimals). | | scale | integer | Number of digits after the decimal point for a numeric column. | | extraPrecision | integer | Overrides the [*Extra Precision*](/reference/configuration-tab/database-schema) added to a numeric column for the specified field. | | htmlFormatted | boolean | Overrides the Salesforce field definition to denote the field is HTML formatted. | ### About ***extraPrecision*** The ***extraPrecision***attribute overrides how many additional precision and scale digits CopyStorm adds to a numeric field when *Strict Decimal Mode* is NOT enabled. \*\*Example:\*\*If a field override such as the below override is provided, the resulting column definition in the database would be “AccountRevenue\_\_c decimal(22,6)” — 18 + 4 precision, 2 + 4 scale. ``` ``` #### **Important Notes:** * If *extraPrecision* is defined for a field, it overrides the system-wide ***Extra Scale*** setting for the specified field. * When using *extraPrecision*, the total precision for a field must NOT exceed the database limit of 38. * *extraPrecision* is ignored when ***Strict Decimal Mode***is enabled. ## Problem: Multipicklist Columns Are Too Wide A configuration like the following will cause all multipicklist type columns in CopyStorm to be 120 characters wide (rather than the Salesforce default of 4099): ``` ``` The tables being overridden can be specified by including an optional “table” parameter. In the following example the Contact table multipicklist columns will be 80 characters, with other tables using 120 characters: ``` ``` ## Problem: String Formulas Are Too Wide A configuration file like the following will cause all string formula type columns in CopyStorm to be 255 characters wide (rather than the Salesforce default of 1300): ``` ``` The table being overridden can be specified using an optional “table” parameter. In the following example the Contact table string formula columns will be 80 characters with other tables being limited to 120 characters: ``` ``` ## Problem: Salesforce Lies About Column Length If Salesforce has a single digit numeric column (e.g. DECIMAL(1)) and an aggregate column SUMs this column, Salesforce will sometimes declare the SUM column with a DECIMAL(1) type. This causes problems when a SUM() is greater than or equal to 10 — without an override CopyStorm will read Salesforce metadata, create a DECIMAL(1) column, and report a “value too large” error when attempting to insert records where the SUM() value is too large. ### Fixing Salesforce Metadata Inconsistencies This configuration will alter the precision of the “AnnualRevenue” column for the “Account” object from DECIMAL(18,0) to DECIMAL(18,3): ``` ``` This configuration will alter the custom decimal field “MonthlyRevenue\_\_c” column for the “Account” object to DECIMAL(24, 4): ``` ``` This example increases the length of the Account.Name database column from 80 to 120 and the Account.Description column to 200: ``` ``` #### **Digits vs. Precision: When to Use Each** * Use ***digits*** for fields that store whole numbers (integers). This maps closely to how Salesforce defines numeric length for non-decimal types. * Use ***precision*** for decimal or floating-point fields. This ensures the appropriate structure is generated in the target database. ### Strict Decimal Mode Behavior There is different behavior for the creation of numeric fields in the CopyStorm database depending on if the *Use Strict DECIMALs* configuration parameter is utilized. For more information on this option, see [here.](/reference/configuration-tab/database-schema) * **Strict Decimal Mode:** CopyStorm will create or update numeric fields in the database using the exact precision and scale as provided by Salesforce. * **Non-Strict Decimal Mode:** CopyStorm will create or update numeric fields in the database with additional precision as denoted by the *Extra Precision* field on the *Database Schema* section of the *Configuration* tab (By default this value is set to 2). ### What to Expect by Database The behavior of *precision* and *scale* can vary depending on the target database. Here is a quick breakdown: | Database Type | Strict Decimal Mode | Non-Strict Decimal Mode | | ------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | SQL Server | Uses the specified *precision* and *scale* | Adds the specified extra digits and ensures a minimum total width of 19 digits | | H2 | Uses the specified *precision* and *scale* | Adds the specified extra digits and ensures a minimum total width of 20 digits | | Oracle | Uses the specified *precision* and *scale* as *NUMBER(precision, scale)* | Utilizes the generic *NUMBER* datatype without enforced limits | | PostgreSQL | Uses the specified *precision* and *scale* | Same behavior as Strict Decimal Mode | | Redshift | Uses the specified *precision* and *scale* as *NUMERIC(precision, scale)* | Adds the specified extra digits and ensures a minimum total width of 19 digits | | Snowflake | Uses the specified *precision* and *scale* | Adds the specified extra digits and ensures a minimum total width of 19 digits | | MySQL | Uses the specified *precision* and *scale* | Adds the specified extra digits and ensures a minimum total width of 20 digits | ## Problem: Completely Ignore Salesforce Columns Sometimes, but rarely, a company desires to completely ignore Salesforce columns and not transfer them or their data into the CopyStorm backup. Most of this time this comes up during a migration where backing up fields associated with an obsolete managed package is desired. Here are a few examples of how to completely ignore fields. * The first *IgnoreField* directive ignores the field Account.MyCustomField\_\_c * The second *IgnoreField* directive ignores all columns added by the Salesforce CPQ package. Note that the special table name default means to apply this rule all tables. Be careful! An *IgnoreField* directive tells CopyStorm to pretend that a column no longer exists in Salesforce and, by default, when a column no longer exists in Salesforce it will be dropped in the CopyStorm database. ``` ``` The options for the *IgnoreField* directive are: * table — name of the Salesforce table containing the field(s). The special name *default* means that the rule will apply to all tables. * name — a regular expression indicating which columns should be ignored. Of course, just a column name also works since this is simple regular expression. Note that this feature was added in CopyStorm 9.47.1. Earlier CopyStorm versions will silently ignore *IgnoreField* directives. # How to Set Table Specific Processing Options There are a number of rare cases in which table-specific processing rules are necessary (please remember, these are rare cases): * The Salesforce metadata API reports fields in a table that are not actually present. * The default LIMIT option for a table always times out. * The Salesforce SOAP API returns an invalid response for an Attachment while the REST API works properly. In all cases CopyStorm supports overriding default processing rules for a table by creating an XML configuration file “TableRuleRegistry.xml” in a CopyStorm configuration directory. The best practice for creating a configuration directory is to: * Create a “config” subdirectory in the directory where .copyStorm files are kept. * This directory should not be inside the CopyStorm installation directory. * Set the “Config Directory” parameter on the CopyStorm Configuration tab in the External Configuration section to point to the newly created config directory. * Store the “TableRuleRegistry.xml” configuration file in the newly created config directory. For other best practices on how to install CopyStorm and your configuration files, please see our [Installation Best Practices](/best-practices/copystorm-directory-structure) article. A quick and dirty solution is to store custom configurations in a subdirectory named “config” in the CopyStorm installation director, but this makes upgrading the CopyStorm installation more difficult: * If CopyStorm.bat is in the directory “C:\bin\CopyStorm” * Create a file named “C:\bin\CopyStorm\config\TableRuleRegistry.xml” ## Task: Prevent Large Attachments from being extracted from Salesforce The following TableRuleRegistry.xml file will cause CopyStorm to exclude Attachment and ContentVersion records where their data size is greater than 1,000,000 bytes: ``` ``` ## Task: Prevent fields from being extracted from Salesforce The following TableRuleRegistry.xml file will cause CopyStorm to exclude the LastViewedDate and LastReferencedDate fields when processing the Salesforce Opportunity table (columns will be created in the database but never populated): ``` ``` The following TableRuleRegistry.xml file will cause CopyStorm to exclude all fields called “LastViewedDate” from tables in the “XYZ” package (columns will be created in the database but never populated): ``` ``` ## Task: Specify exactly which fields are extracted from Salesforce The following TableRuleRegistry.xml file will cause CopyStorm to only include the “Name” field when processing the Salesforce Opportunity table (other columns will be created in the database but never populated): ``` ``` Note that: * When an IncludeField is specified any ExcludeField rules are ignored. * The following fields are always copied, even if they are in a configuration directive: * SystemModStamp * CreatedDate * LastModifiedDate * LoginTime * isDeleted ## Task: Exclude Formulas from CopyStorm tables Default CopyStorm behavior is to backup formula columns just like any other column. To change this behavior, the “excludeFormulas” directive will cause CopyStorm to ignore formulas on a table. Note that the database columns corresponding to Salesforce formulas will still be created but will always have null values. The following TableRuleRegistry.xml file will cause CopyStorm to exclude formulas for the Opportunity table. ``` ``` Starting with CopyStorm version 8.37.1 the default formula exclusion rule can be set for all tables by setting the “excludeFormulas” attribute on the table “default”. The “default” table does not correspond with a table name in Salesforce or the CopyStorm database, but is used to specify global table processing options. The following TableRuleRegistry.xml file will cause CopyStorm to exclude formulas for all tables (unless there is a rule overriding the global default for individual tables): ``` ``` ## Task: Change the default SOQL LIMIT value used for a table when reading from Salesforce The following TableRuleRegistry.xml file will cause CopyStorm to read Opportunity records 17,777 at a time: ``` ``` ## Task: Change the Field used to determine if a record has been modified Default CopyStorm behavior is to use the systemmodstamp field to determine when a record has been modified. This ensures that all changes are mimicked in CopyStorm, but occasionally a site will want to use the LastModifiedDate column instead. The following TableRuleRegistry.xml file will cause CopyStorm to use the “LastModifiedDate” column to determine if “Account” records have changed: ``` ``` Since this would be tedious to apply to all tables, another option is to change the global default for all tables. Note that if a table does not have the specified timestamp column, CopyStorm will automatically adjust to the most pertinent column contained in the table. The following TableRuleRegistry.xml file will cause CopyStorm to use the “LastModifiedDate” to determine if any record has changed: ``` ``` ## Task: Force CopyStorm to use a specific database table name Default CopyStorm behavior is to create database tables with names matching the corresponding table in Salesforce. To override the database name for a specific Salesforce table, use the “databaseTableName” parameter. The following TableRuleRegistry.xml file will cause CopyStorm to store “Account” records in a table named “AccountMaster”: ``` ``` ## Task: Add database specific table creation options Default CopyStorm behavior is to use the default RDBMS options. To specify non-default options for table creation use the “CreateTableRule” configuration directive. The following TableRuleRegistry.xml file will cause CopyStorm to use specific creation options in MySQL and SQL/Server: ``` ``` If the database type matches the “database” parameter the specified option will be appended to the generated CREATE TABLE statement. For example: CREATE TABLE Account(id CHAR(18) PRIMARY KEY, name NVARCHAR(80) NULL, …) ENGINE=ISAM; To apply a rule to all tables, set the “name” of the TableRule to “default”. ## Task: Limit the amount of time CopyStorm will spend on a table Default CopyStorm behavior is to process each table until all records are backed up. Setting the “maxRuntime” parameter will cause CopyStorm to stop processing a table after an amount of time has elapsed. The maxRuntime is specified in seconds. The following TableRuleRegistry.xml file will cause CopyStorm to stop processing Attachment records and continue with the next table in the backup set after 10 minutes has elapsed: ``` ``` ## Task: Set table specific BULK API processing parameters Default CopyStorm behavior is for all table to use the Bulk API parameters specified on the CopyStorm Advanced tab. Several of these parameters can be overridden on a per-table basis. The following TableRuleRegistry.xml file will cause CopyStorm to use table-specific Bulk API rules for the Account table: * Switch to the Bulk API after 5000 records have been processed by the SOAP API. * Read 100,000 records per Bulk API transaction. ``` ``` ## Task: Force the REST API to be used for a table The following TableRuleRegistry.xml file will cause CopyStorm to always use the REST API for the Attachment table: ``` ``` ## Task: Change the Record Deletion Policy for a table Default CopyStorm behavior is to use the same record deletion policy on all tables, specified by the “Delete Older Than” option on the Advanced tab. The following TableRuleRegistry.xml file will cause CopyStorm to never remove deleted records from the Attachment table (overriding the “Delete Older Than” global parameter): ``` ``` ## Task: Force a table to backup by ID Order Default CopyStorm behavior is to back up all columns in timestamp order. For rare cases where a table does not contain a timestamp column, a table can be forced to backup in ID order. In nearly all cases, this option is only used internally by CopyStorm for Salesforce system tables that do not follow the normal rules. The following TableRuleRegistry.xml file will cause CopyStorm to backup the PartnerNetworkRecordConnection table by ID (this is an example of a Salesforce system table that cannot be backed up in timestamp order): ``` ``` ## Task: Force a table to be TRUNCATED before each copy The following TableRuleRegistry.xml file will cause CopyStorm to delete all data in the “PartnerNetworkRecordConnection” before starting each copy: ``` ``` ## Task: Force a table to backup using a Timebox Window A number of tables in Salesforce do not support sorting by their timestamp column, and other tables where a user without the ViewAllData permission will get stuck. In these extremely rare cases, CopyStorm can be forced to backup a table using a sliding timebox approach. The following TableRuleRegistry.xml file will cause CopyStorm to backup the LoginEvent table using a sliding timebox on the EventDate field: ``` ``` This XML rule means: Use a sliding 90 day window when backing up LoginEvent. Start the first search on 1-Jan-2014 or the most recent EventDate in the LoginEvent backup table. The system will backup LoginEvent entries in roughly the following order: 1-Jan-2014 to 31-Mar-2014 1-Apr-2014 to 30-Jun-2014 1-Jul-2014 to 30-Sep-2014 etc… until the current date is reached. Subsequent backups will start the first timebox at the most recent EventDate in the LoginEvent backup table. ## Summary A single TableRuleRegistry.xml file can contain multiple rules and any or all features can be specified for a specific table. However, this is a bad practice unless instructed to by Capstorm. The built-in rules have been validated on many Salesforce instances and are generally optimal. # How to Set Database Specific Initialization Options In certain configurations it may be necessary to run specific SQL statements or set database specific options each time a connection to a database is established. ## Oracle The configuration file “Oracle.xml” contains parameters that control connections made to Oracle. The following configuration directives are supported: * InitSQL — specify a line of SQL that will be run each time a new connection to Oracle is established. There is no limit to the number of InitSQL elements. The following Oracle.xml file will cause CopyStorm to run the command: * `alter session set ‘_optimizer_rownum_pred_based_fkr’=FALSE` Oracle.xml: ``` ``` # How to Transform Field Data During a Backup CopyStorm supports dynamic transformation of field data after reading it from Salesforce and before it is written to the target database. This article describes how to add a new custom transformation to CopyStorm. ## History of This Feature This issue first arose when a customer wanted to scramble Salesforce data for certain fields before it was written to the target database. Since the customer needed total control over the data manipulation process, an extension was added to CopyStorm to dynamically add field transformations. This allows you to write custom code to perform operations like: * Scramble email addresses – instead of replicating plain-text email addresses, obfuscated emails could be replicated. * Scramble sensitive data such as social security numbers or Contact health information. Most customers do not require this feature — it is only used when Salesforce data needs to be modified prior to being added to a local relational database. As CapStorm’s data recovery features will recover the exact data that is stored in the database, it is **always** best practice to write unmasked data to the database. ## Feature Overview The process of adding a new data transformer is not difficult — all that is required are the following steps: 1. Write a Java class that implements the Capstorm interface com.aslan.sfdc.extract.transform.ITransformer. 2. Add a TransformerRegistry.xml file to CopyStorm’s config directory to register the new transformer with CopyStorm. 3. Update the TableRuleRegistry.xml file to enable the new transformer for the appropriate fields. The rest of this article describes the development and deployment process for adding a new data transformer to CopyStorm that reverses any input string. Example: The transformer in this article will turn “Capstorm” into “mrotspaC”. ## Step 1: Write a Java Class Implementing the Transformer The following code is an example data transformer implementation. After creating a new data transformer, package it in a jar and store the jar file in the “lib” subdirectory of the CopyStorm installation directory. ``` package com.capstorm.extension; import org.apache.commons.lang3.StringUtils; import com.aslan.sfdc.extract.transform.ITransformer; public class ReverseTransformer implements ITransformer { /** * Take a string and return the string with the letters reversed. * *@param table - name of the Saleforce table containing the value *@param column - name of the Salesforce column containing the value *@param columnWidth - the number of characters in the column *@param recordId - the unique Saleforce record id containing the value *@param value - the values whose letters are to be reversed. * *@return the value with its letters reversed */ @Override public String transform(String table, String column, int columnWidth, String recordId, String value) { if( null != value ) { value = StringUtils.reverse(value); } return value; } } ``` ## Step 2: Register the Transformer with CopyStorm CopyStorm discovers available transformers by looking for a file named TransformerRegistry.xml in the CopyStorm config directory. The following TransformerRegistry.xml will cause CopyStorm to load the “Reverse” transformer: ``` ``` The Transformer directive adds a new custom transformer using the following attributes: * name — the friendly name used to apply the transformer to a particular Salesforce field. * class — the full name of a class that implements the ITransformer interface. ## Step 3: Attach a Transformer to a CopyStorm Field CopyStorm discovers field transformations by looking for FieldTransform elements in the TableRuleRegistryFile.xml configuration file in CopyStorm’s config directory. The general format of this file is described [in this article](/reference/configuration-files/how-to-set-table-specific-processing-options). The following TableRuleRegistry.xml will cause CopyStorm to reverse the letters in the LastName for all Contacts before writing the LastName to the database: ``` ``` FieldTransform configuration directives have the following options: * name — the name of a field in the corresponding Salesforce table. * transformer — the name of a registered transformer that will be applied to the field data. This is the same as the name specified in Step 2. ## My Transformer Does Not Work. What Should I Do? If your custom transformer is having no effect on data then the problem is likely in the new XML configuration directives. To verify that your entry to TableRuleRegistry.xml is correct, use one of the built-in CopyStorm transformers. Built-in transformers include: * Scramble — replace each letter with a different letter, replace each number with a different number. * Blank — return a blank string. If CopyStorm’s built-in transformer works then the problem is either in your TransformerRegistry.xml file or in your code. Put some debugging in your transformer to see if it is ever being called. * If it is not being called, the problem is in the TransformerRegistry.xml file. * If it is being called, then fixing the code is up to you! # How to Control Field History Tracking CopyStorm supports the tracking of nearly all field changes by checking the “Track Field History” box on the Advanced tab. When field history is enabled, default behavior is to track updates to EVERY field in EVERY table except for a handful of Salesforce auditing fields. If the default behavior captures more data than your organization prefers, this article explains how to limit what is tracked. ## Background Fine control over field tracking is managed by a configuration file named FieldHistoryRecordFactory.xml. The file can be placed in either the “config” subdirectory of the CopyStorm installation or in the directory indicated in the CopyStorm configuration file. The built-in configuration file has rules excluding field tracking for a number of Salesforce audit fields and all formula columns. It looks like: ``` ``` If you create your own configuration file, model it after the built-in file without repeating any of the built-in rules. Like CopyStorm table selection, there are two basic approaches to controlling which fields are tracked: * Telling CopyStorm which fields should be tracked. * Telling CopyStorm which fields should NOT be tracked. If CopyStorm is told which fields to track the rules indicating which fields should NOT be tracked are ignored and only fields CopyStorm is explicitly told to track will be included. ## Include Specific Tables To only track field history on specific tables, include the IncludeTable configuration directive. The following FieldHistoryRecordFactory.xml will limit field history to the Account, Contact, and MyCustomObject\_\_c tables: ``` ``` ## Exclude Specific Tables To track fields in all tables EXCEPT for a few then use the ExcludeTable configuration directive. The following FieldHistoryRecordFactory.xml will exclude ApexPages, ApexTriggers, and MyCustomObject\_\_c from field history tracking: ``` ``` ## Include Specific Fields in a Table The following FieldHistoryRecordFactory.xml will cause CopyStorm to only track the “Name” field on “Account” and the “FirstName”/”LastName” fields on “Contact”: ``` ``` ## Exclude Specific Fields in a Table The following FieldHistoryRecordFactory.xml will cause CopyStorm to not track the “WebSite” field on “Account”, and to not track the “Age”/”Birthday” fields on “Contact”: ``` ``` ## Track Formula Value Changes Default CopyStorm behavior is to not track changes to formula column values. The following FieldHistoryRecordFactory.xml will enable formula value history tracking: ``` ``` # How to Change License Servers If your site does not permit communication with the default Capstorm license server located at the configuration file “Capstorm.xml” allows overriding the endpoint. An example that changes CopyStorm to use a http (non-SSL) endpoint for license management is: ``` ``` There are a number of other elements that can appear in a Capstorm.xml file, but you should not change without being instructed to by Capstorm. However, if you want to confuse your colleagues you can change CopyStorm’s background color: ``` ``` # How to Solve UNKNOWN\_EXCEPTION Errors ## How to Solve UNKNOWN\_EXCEPTION Errors On rare occasions a site may receive an UNKNOWN\_EXCEPTION from CopyStorm. In CopyStorm’s log, the error will look like: 10-Feb-17 09:02:11: \[UnexpectedErrorFault \[ApiFault exceptionCode=’UNKNOWN\_EXCEPTION’ exceptionMessage=’An unexpected error occurred. Please include this ErrorId if you contact support: 303701882-51722 (1187285824)’ extendedErrorDetails='{\[0]}’ This error is generated by Salesforce’s server when it encounters a problem that Salesforce code does not know how to handle. Most of the time the issue will resolve itself after a couple days as Salesforce fixes their own internal problems. If the problem does not clear up automatically then this article may help you resolve the issue. In Capstorm’s experience, to get Salesforce support to pay attention to UNKNOWN\_EXCEPTION issues a problem has to be reproduced in an application supplied totally by Salesforce. When this error is being given by Salesforce we need to: * Isolate the SOQL query in CopyStorm that is causing the problem. * Run the query in the Salesforce console to prove that the query is causing the issue. * Submit a case to Salesforce. ## Step 1: Isolate the SOQL Query in CopyStorm 1. Start CopyStorm from a command line window with the “-debug” command line switch. 1. CopyStorm.bat -debug 2. sh CopyStorm.sh -debug 2. On the Advanced tab, select “Debug Mode” under “Misc Options”. 1. You may have to select “Show All Options” to see the “Debug Mode” option. 3. Start a copy. The SOQL query causing the issue will appear in the command line window. 4. Copy the SOQL and save it to a file. ## Step 2: Run the SOQL Query in the Salesforce Console 1. Login to the Salesforce GUI. 2. Choose “Developer Console” in the menu that appears when you click on your name. 3. Paste the SOQL query into the editor. 4. Press the execute button. 5. Copy the error text and save it. 1. The error should appear in the lower part of the screen. ## Step 3: Submit a Support Case to Salesforce * Describe what you did in Step 3. * Include the problem SOQL and the error text from Step 3. First level support can rarely solve this sort of issue and you should expect a request to grant access to your system so Salesforce engineers can reproduce the error. ## Why Does This Type of Error Happen? Though we do not see this error often, it has appeared a few times when Salesforce index corruption of their Oracle database caused the issue. Truthfully we cannot know the precise root cause unless Salesforce exposes it — to any application it is only exposed as a type of error that Salesforce does not expect to occur. # How to Solve OPERATION\_TOO\_LARGE Errors ## How to Solve OPERATION\_TOO\_LARGE Errors If CopyStorm is reporting an OPERATION\_TOO\_LARGE error, it will be included in the CopyStorm log. The error will look like this: 09:02:11: \[UnexpectedErrorFault \[ApiFault exceptionCode=’OPERATION\_TOO\_LARGE’ exceptionMessage=’exceeded 20000 distinct ids’ extendedErrorDetails='{\[0]}’ ] ] Before diving into the root cause, this is the most common solution: * Grant the ViewAllData privilege to the Salesforce credentials being used with CopyStorm. ## Root Cause The root cause of this exception can be found in the Salesforce description of the error: * `OPERATION_TOO_LARGE` The query has returned too many results. If certain queries are run by a user without the “View All Data” permission and many records are returned, the queries require sharing rule checking. For example, consider queries that are run on objects, such as [Task](https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/sforce_api_objects_task.htm), that use a polymorphic foreign key. These queries return this exception because the operation requires too many resources. To correct, add filters to the query to narrow the scope, or use filters such as date ranges to break the query up into a series of smaller queries. You can find the Salesforce documentation in the [Core Data Types](https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/sforce_api_calls_concepts_core_data_objects.htm) section. Almost all CopyStorm sites will want to use Salesforce credentials with the ViewAllData permission because without this permission all data will not be backed up. In particular, no metadata will be backed up because the API call used to fetch metadata requires the ViewAllData permission. ## I Need a Timebox Approach Because ViewAllData is Not Allowed If your organization does not want to give your CopyStorm backup user the ViewAllData permission and an incomplete backup is OK, you can configure CopyStorm to backup data using a sliding timebox. For example, to backup the Note table using a 60 day sliding timebox, the following TableRule must be added to your site’s [TableRuleRegistry.xml](/reference/configuration-files/how-to-set-table-specific-processing-options) file: ``` ``` This XML rule means: * Use a sliding 60 day window when backing up Note. * Start the first search on 1-Jan-2001. * Start subsequent searches at the timestamp on the most recent Note record in the backup. The system will backup Note records in roughly the following order: 1-Jan-2001 thru 28-Feb-2001 1-Mar-2001 thru 30-Apr-2001 1-May-2001 thru 30-Jun-2001 etc… until the current date is reached. # How Does CopyStorm Search for Configuration Files? There are a number of XML configuration files that can be used to override default CopyStorm behavior. This article describes the approach CopyStorm uses to find configuration files. When CopyStorm searches for a configuration file, it looks in four places: 1. Configuration files embedded within the CopyStorm application. 1. CopyStorm always looks here first. 2. Configuration files in the “config” subdirectory of the CopyStorm installation. 1. CopyStorm always looks here. 3. Configuration files in the directory specified by the Configuration tab “Config Directory” option within the “External Configuration” section. 4. Configuration files in the directory specified by the “-config” command line argument. 1. CopyStorm looks here ONLY IF the Advanced tab setting is blank. If a configuration file is found in multiple locations, all are loaded in the order listed above. Later loaded files will override any configuration from earlier files. Documentation on available XML configuration files can be found on [this page](/reference/configuration-files). ## Best Practices Capstorm recommends the following best practices: * Keep each CopyStorm configuration file in its own directory OUTSIDE of the CopyStorm installation directory. * This will make the CopyStorm upgrade process simpler. * If you need custom XML configuration files then: * Store them in the same directory as the corresponding .copyStorm configuration file. * Set the Configuration tab “Config Directory” parameter in your .copyStorm file to point to the directory where the .copyStorm file is located. # How to Fix Incorrect String Value Errors for MySQL Though MySQL claims support for all UTF-8 characters, experience indicates that not all MySQL variants support the full extended 4 byte UTF-8 character set. When MySQL fails to recognize an extended UTF-8 character and error like this will be displayed: * `java.sql.BatchUpdateException: Incorrect string value: ‘\xF0\x9F\x98\x8A’ for column ‘CommentBody’ at row 48` To get around this issue, you can configure CopyStorm to replace unsupported 4 byte UTF-8 characters with a substitution. Note: The extended 4 byte UTF-8 character set contains like 100 variations of a smiley face, trees, hamburgers, etc… ## How to Define Character Mappings Character mappings are defined in a “UTF8Tool.xml” configuration file that can be placed in either the “config” subdirectory of the CopyStorm installation or in a custom config directory you have created. The following UTF8Tool.xml will cause CopyStorm to replace the 4 byte UTF-8 characters for two smiley faces to be converted to their more old fashioned counterparts: ``` ``` This rule will cause any text containing the specified smiley face characters to have the characters be replaced by “:)” or “:(” before data is written to the database. ## How to Convert Hex Strings to UTF Characters Since MySQL reports UTF-8 problems using a hex byte sequence, you will need a way to convert hex byte sequences to a character suitable for XML. Capstorm has used the tool located at: * To use the tool enter the hex digits from MySQL into the box labeled “UTF-8 Code” (e.g. 20 E2 88 9A) and press the Convert button. Copy the generated unicode character into the “from” parameter in UTF8Tool.xml. Note that you can use the regular UTF-8 character in the “from” parameter rather than the decimal or hex representation. # How to Enable Oracle’s EXTENDED Varchar2 Option Oracle 12c added support for extending the allowed length of VARCHAR2 columns from 4000 to 32,767 bytes but made the discovery of whether this feature is enabled require privileges that a normal Oracle user is not granted. This article explains how to tell CopyStorm that the Oracle feature “max\_string\_size=EXTENDED” is enabled without requiring special Oracle permissions. For your reference, if an Oracle user has enough privileges then a simple query can be used to determine if the EXTENDED Varchar2 feature is enabled. * `SELECT name,value FROM V$PARAMETER WHERE name=’max_string_size’` It is likely that your account will return a “Table or View not Found” when running this query because of your limited permissions. To inform CopyStorm that the EXTENDED feature is enabled simply create a file named Oracle.xml and drop into the your private CopyStorm configuration directory specified on the Advanced tab or into the directory \[CopyStormRootDir]/config. The content of the Oracle.xml file should be: ``` ``` Restart CopyStorm and future VARCHAR2 columns will be up to 32,764 bytes in length. You may wonder why the limit is 32,764 rather than 32,767. The reason is that Salesforce defaults to a UTF-8 Extended character set and each character can take up to 4 bytes. # How to Override Snowflake staging location Internally, CopyStorm replicates data to Snowflake by creating a CSV file of the records being backed up and uploading the CSV file to Snowflake. It may be required to change this staging location to meet business requirements. CopyStorm allows customers to specify a different Snowflake stage by following this process: Create a xml file named “SnowflakeDatabaseBuilder.xml” Then place the contents below into the xml file. ``` ``` You will need to replace “clientProvidedStagePath” with the Snowflake stage’s path, then place this file into CopyStorm/config directory and relaunch CopyStorm. Note: This feature is available in CopyStorm versions 10.56.5 and above. # How to Set Application Initialization Options ## Background **Note:** This feature is available in Capstorm application versions 10.56.6 and above. Capstorm applications allows you to add supplementary Java options. These options serve to dynamically alter the operational parameters of the application prior to its initiation. One frequent utilization, which serves as an example throughout this article, is the facilitation of user-driven control over memory allocation for any of Capstorm’s applications. This capability gives users the control to finely calibrate the memory resources dedicated to the operation of Capstorm’s applications. ## Configuration File Creation and Placement **Note:** Application must be restarted for these configuration files to be applied. ### **Step 1 – File Creation and initial template setup:** 1. First create a XML file named “AppLoader.xml” ### **Step 2 – File Placement:** Two recommended placement options: * Option 1: If you followed CopyStorm’s Best Practice Directory Structure then we recommend placing this configuration file in CapstormConfig directory. * [copystorm-directory-structure](/best-practices/copystorm-directory-structure) * Option 2: Inside of one of Capstorm’s application installation directory there should be a directory called “config”. You can also place this configuration file in the “config” directory. ## How to Customize Application Memory **Note:** For versions older than 10.56.6 please visit: [out-of-memory-java-heap-space-error](/frequently-asked-questions/problems-solutions/out-of-memory-java-heap-space-error) on how to configure memory settings. * Once you have completed “Configuration File Creation and Placement” steps you can simply copy the snippet below into the AppLoader.xml configuration file. * The example below shows how to set the min and max memory allocations to 300 and 1000 megabytes, respectively. ``` ``` ## How to Add Custom Java Properties The AppLoader also allows for custom Java Properties. Below is an example of setting the HTTPS Protocols to TLSv1.1 and TLSv1.2. ``` ``` # How to Change Schema Initialization Options While commonly used schema options within CopyStorm are directly exposed in the CopyStorm GUI, a number of schema initialization parameters can only be provided by providing CopyStorm with a *JDBCDatabaseBuilderConfig.xml* file. To use the options detailed in this page, create a *JDBCDatabaseBuilderConfig.xml* file and copy it to one of the CopyStorm [Configuration File Directories](/reference/configuration-files/how-does-copystorm-search-for-configuration-files). ## Capitalize Column & Table Names The following XML will cause CopyStorm to create a Salesforce data table and column names in ALL CAPS. ``` ``` ## Remove Table & Column Postfixes The following XML will cause CopyStorm to remove Salesforce-standard postfixes from database table and column names. ``` ``` # Command Line CopyStorm provides a command line interface to enable CopyStorm to run under Windows Task Scheduler or CRON on Linux. Begin by creating a CopyStorm [configuration file](/reference/configuration-files). The following example Windows command line will run CopyStorm where: * CopyStorm is installed in C:\CopyStorm. * CopyStorm’s configuration file name is DemoCredentials.copyStorm. * CopyStorm’s configuration file is located in C:\CopyStormDemo. * The log file is redirected to a text document named C:\users\Mary\LogFile.txt. * This is optional. ``` CopyStorm.bat -run ../CopyStormDemo/DemoCredentials.copyStorm >c:/users/Mary/LogFile.txt ``` A similar command line for Linux or Mac will look like: ``` sh CopyStorm.sh -run ../CopyStormDemo/DemoCredentials.copyStorm >/home/mary/LogFile.txt ``` ## Command Line Reference | Switch | Parameter | Description | | --------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -sftype | Sandbox | Production | | -sfuser | Username | Override the Salesforce username in the configuration file. | | -sfpassword | Password | Override the Salesforce password in the configuration file. | | -sftoken | Security Token | Override the Salesforce security token in the configuration file. | | -dbconnection | Connection String | Override the database connection string in the configuration file. | | -dbuser | Username | Override the database username in the configuration file. | | -dbpassword | Password | Override the database password in the configuration file. | | -dbschema | Schema Name | Override (or set) the database schema value in the configuration file. | | -proxyHost | Host Name | Override the proxy server in the configuration file. Example: proxy.capstorm.com | | -proxyPort | Port Number | Override the port number used by the proxy server. | | -proxyUser | Username | Override the username used to authenticate with the proxy server. | | -proxyPassword | Password | Override the password used to authenticate with the proxy server. | | -config | Directory | Search this directory for XML configuration overrides. Configurations in this directory will override those in the “config” subdirectory in CopyStorm’s installation directory. | | -apiTraceFile | Filename | Record all messages to/from Salesforce in this file. Note that this will generate a very large file and is typically only useful when debugging problems with Salesforce. | | -debug | | Write verbose debugging information, including SOQL statements sent to Salesforce, to stderr. | | -modifiedSince | | Set the of the Modified Since parameter, overriding any value in the configuration file. | | -modifiedThru | | Set the of the Modified Thru parameter, overriding any value in the configuration file. | | -includedTables | | Set list of tables to include in the backup, overriding any value in the configuration file. | | -excludedTables | | Set list of tables to exclude from the backup, overriding any value in the configuration file. | | -forceTruncate | | Truncate each table before it is backed up (creates a fresh copy of each table) | ## Running a Snapshot Policy Headlessly [Snapshot policies](https://learn.capstorm.com/reference/archive-tab) can be executed from the command line without opening the CopyStorm UI. Use the `-snapshot` switch in place of `-run`: **Windows:** ``` CopyStorm.bat -snapshot "My Snapshot Policy" -snapshotThreads 5 -snapshotSearchBatchSize 5000 -snapshotWriteBatchSize 5000 -snapshotScanRecordsFrom "2023-01-01" -snapshotScanRecordsThru "2024-01-01" >C:\logs\snapshot.txt ``` **Linux/Mac:** ``` sh CopyStorm.sh -snapshot "My Snapshot Policy" -snapshotThreads 5 -snapshotSearchBatchSize 5000 -snapshotWriteBatchSize 5000 -snapshotScanRecordsFrom "2023-01-01" -snapshotScanRecordsThru "2024-01-01" >/home/jake/snapshot.txt ``` | Switch | Parameter | Description | | ------------------------ | ----------- | ------------------------------------------------------------------------ | | -snapshot | Policy Name | Name of the snapshot policy to run. | | -snapshotThreads | Count | Number of objects to snapshot concurrently. | | -snapshotSearchBatchSize | Batch Size | Number of records read from the database per search batch. Default: 1000 | | -snapshotWriteBatchSize | Batch Size | Number of records written to the database per batch. Default: 1000 | | -snapshotScanRecordsFrom | Date | Limit snapshot to records modified on or after this date (YYYY-MM-DD) | | -snapshotScanRecordsThru | Date | Limit snapshot to records modified on or before this date (YYYY-MM-DD) | ## CopyStorm Exit Codes In command line mode CopyStorm returns these exit status codes: * 0 = the run completed normally with no exceptions. * 1 = the run completed but had one or more recoverable errors. * 2 = the run may have completed but there are exceptions that need investigation. * 2 = credential or license error — one of: * Salesforce or database credentials are wrong. * Your CopyStorm license has expired # CopyStorm/Medic CopyStorm/Medic is a collection of tools for maintaining, validating, and repairing a CopyStorm database. CopyStorm/Medic can be downloaded from [www.capstorm.com](https://www.capstorm.com/). Tools provided with CopyStorm/Medic include: | Tool | Tool Name | Tool Path | Description | | ----------------------------------------------------------------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Log Manager](/reference/copystorm-medic/log-manager) | |

Other Tools
→Log Manager

| View, manage, and export the status of a historical CopyStorm run. | | [Fix IsDeleted](/reference/copystorm-medic/fix-isdeleted) | fixDeleted |

Other Tools
→Data Synchronization
→Fix Deleted

| If a record is deleted in Salesforce and the Salesforce recycle bin entry is cleared before CopyStorm runs, CopyStorm will not know that the Salesforce record has been deleted. The Fix IsDeleted tool finds records in CopyStorm that are no longer in Salesforce and marks them as deleted in CopyStorm. | | [Refresh Formulas](/reference/copystorm-medic/refresh-formulas) | refreshFormulas |

Other Tools
→Data Synchronization
→Refresh Formulas

| The Refresh Formulas tool updates all formula columns in a CopyStorm database with fresh data from Salesforce. This tool is needed when a formula definition changes in Salesforce. | | [Scan Timestamps](/reference/copystorm-medic/scan-timestamps) | scanTimestamps |

Other Tools
→Database Validation
→Scan Timestamps

|

The Scan Timestamps tool does an exhaustive comparison of Salesforce versus CopyStorm record modification timestamps. Capstorm provides this tool for two reasons:

  • To provide assurance that CopyStorm is copying all records properly.
  • To assist in tracking down “after update” triggers on a table that updates the source table.
| | [Refresh Column(s)](/reference/copystorm-medic/refresh-columns) | refreshColumns |

Other Tools
→Data Synchronization
→Refresh Columns

| The Refresh Column(s) tool updates any column in a CopyStorm database with fresh data from Salesforce. It’s primary purpose is to update a particular formula column but can be used for any column type. | | [Table Editor](/reference/copystorm-medic/table-editor) | tableEditor |

Other Tools
→Rarely Used
→Table Editor

| Manually truncates or deletes CopyStorm tables. Note that CopyStorm will recreate the tables the next time it runs. | | [Count Records](/reference/copystorm-medic/count-records) | countRecords |

Other Tools
→Database Validation
→Count Records

| Count the number of records in a Salesforce table and the corresponding CopyStorm table and compare the results. | | [Find Dirt](/reference/copystorm-medic/find-dirt) | |

Other Tools
→Salesforce Data Quality
→Find Dirt

| The Find Dirt tool finds data in Salesforce fields that should not be possible and does not match the field definition. | | [Permission Checker](/reference/copystorm-medic/permission-checker) | findPermissionErrors |

Other Tools
→Database Validation
→Permission Checker

| The Permission Checker tool discovers tables and fields that are defined in Salesforce but not visible to the current Salesforce user. | | SQL Runner | |

Other Tools
→Rarely Used
→SQL Runner

| Can be used to run any SQL statement against the CopyStorm database. This is not a substitute for a fully-featured tool (e.g. MySQL Workbench) but is useful when a fully-featured tool is not available. | | Fix Base64 | fixbase64 |

Other Tools
→Data Synchronization
→Fix Base64

| Find all null base64 columns in the CopyStorm database and update with the latest value in Salesforce. | | Base64 Sanity Checker | base64LinkChecker |

Other Tools
→Salesforce Data Quality
→Base64 Sanity Checker

| Discover Attachment/Content/etc. base64 columns where Salesforce returns an error when attempting to read the associated data. | | Purge Orphan base64 | purgeOrphanBase64 |

Other Tools
→Data Synchronization
→Purge Orphan Base64

| When large Base64 field values are added to the CopyStorm database, and chunking of that data is enabled, then those field values are broken down into segments that span an auxiliary table called CopyForceTableFieldChunks. It is not possible for the database to do a CASCADING delete on the CopyStorm tables to purge those data chunks when a record is deleted so this tool is provided to purge those orphaned data chunk records. | | Image Scanner | imageScanner |

Other Tools
→Data Synchronization
→Image Scanner

| This tool scans for Salesforce images not stored in the CopyStorm Database. Tool scans for columns in selected tables which contain HTML (RichText) data. When HTML contains a reference to an image the image will be read and stored in the CopyStorm database. | | [Fix DocLinks](/reference/copystorm-medic/fix-contentdocumentlinks) | fixDocLinks |

Other Tools
→Data Synchronization
→Image Scanner

| The Fix DocLinks tool repairs records in the Salesforce ContentDocumentLink table. The Salesforce ContentDocumentLink table has query restrictions that make it impossible to efficiently perform an 100% accurate incremental backup | | [Base64 Export](/reference/copystorm-medic/base64-export) | base64Export |

Other Tools
→Data Synchronization
→Export Base64 Objects to File System

| The Base64 Export tool converts document records in the CopyStorm database into files in a file system. | | [Delete Salesforce Records](https://learn.capstorm.com/reference/copystorm-medic/delete-salesforce-records) | deleteTool | Other Tools → Rarely Used → Delete Salesforce Records | Identifies, audits, and deletes a hierarchy of related Salesforce records. Supports immediate API deletion, Data Loader file generation, and Python script generation. | ## Running CopyStorm/Medic in UI Mode CopyStorm/Medic can be started as a graphical application by running CopyStormMedic.bat (or “sh CopyStormMedic.sh” on Linux). The UI mode is often the only way a site will use the tool, but it can also run as a batch job (details later in this article). The CopyStorm/Medic UI is subdivided into several tabs — the Main tab contains Salesforce and CopyStorm connection credentials and other tabs contain individual CopyStorm/Medic tools. Like other Capstorm applications, a CopyStorm/Medic configuration can be saved/restored to/from an encrypted text file. Saved CopyStorm/Medic configuration files can be used to run CopyStorm/Medic as a batch job. ![](/files/v2FXcOHhVYF1awvlnpHo) ## Running CopyStorm/Medic in Batch Mode After saving a CopyStorm/Medic configuration file, it can be used to launch CopyStorm/Medic in batch mode. The syntax for starting CopyStorm/Medic in batch mode is: * Windows: “CopyStormMedic.bat -tool ToolName -run configFile.CopyStormMedic” * Linux: “sh CopyStormMedic.sh -tool ToolName -run configFile.CopyStormMedic” The value for ToolName must be selected from the table at the beginning of this article. For example, to run the Fix isDeleted tool on Windows: * CopyStormMedic.bat -tool fixDeleted -run configFile.copyStormMedic ## Command Line Reference | Switch | Parameter | Description | | -------------- | ----------------- | --------------------------------------------------------------------------------------- | | -sftype | Sandbox | Production | | -sfuser | Username | Override the Salesforce username in the configuration file. | | -sfpassword | Password | Override the Salesforce password in the configuration file. | | -sftoken | Security Token | Override the Salesforce security token in the configuration file. | | -dbconnection | Connection String | Override the database connection string in the configuration file. | | -dbuser | Username | Override the database username in the configuration file. | | -dbpassword | Password | Override the database password in the configuration file. | | -dbschema | Schema Name | Override (or set) the database schema value in the configuration file. | | -proxyHost | Host Name | Override the proxy server in the configuration file. Example: proxy.capstorm.com | | -proxyPort | Port Number | Override the port number used by the proxy server. | | -proxyUser | Username | Override the username used to authenticate with the proxy server. | | -proxyPassword | Password | | | -apiTraceFile | filename | Write the entire conversation with the Salesforce API to a file (the file may be huge). | # Log Manager The CopyStorm/Medic Log Manager tool supports the viewing, management, and exporting of the results of previously run CopyStorm jobs. Starting with version 7.\*.\*, CopyStorm stores the status of each job into tables in the CopyStorm target database. The CopyStorm/Medic Log Manager is a convenient tool for viewing, managing, and exporting the contents of these tables. In addition to Log Manager, regular RDBMS tools can be used to access CopyStorm’s logs. Full documentation for the CopyStorm/Medic Log Manager tool can be found [here](/reference/history-tab). The database tables used for RDBMS access to CopyStorm and CopyStorm/Medic logs can be found [here](/frequently-asked-questions/how-does-copystorm-work/does-the-copystorm-database-have-logging-tables). # Notifications The Notifications Tab contains parameters used to control who receives a status email when a CopyStorm/Medic job runs and under what circumstances email is sent. CopyStorm/Medic supports email notifications via SMTP or via a Salesforce SOAP API call. ## SFDC Notifications ![](/files/zVLUCD03QvsjCxuFfF7t) | Parameter Name | Required | Description | | -------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Send Email To | Yes | Notification emails will be sent to this white space separated list of email addresses. | | Subject | Yes |

The subject line used for notification emails. Though it can be any text you want, there are a few special text values that will be replaced dynamically before a message is sent:

  • “{status}” will be replaced by “OK” or “ERROR” based on the outcome of the job.

    • If there are any exceptions encountered then “ERROR” will be substituted, otherwise “OK” will be used.
  • “{application}” will be replaced by “CopyStorm/Medic”.

  • “{file}” will be replaced by the base name of the CopyStorm/Medic configuration file used by the job.

    • For example, the configuration file “C:\capstorm\config\myCopyStormMedicConfig.copyStormMedic” will be replaced with “myCopyStormMedicConfig.copyStormMedic”.
| | Send Notification On | No |

Determines when CopyStorm/Medic will send a notification email:

  • Success: If checked, an email will be sent if a job works perfectly (no exceptions).
  • Exception(s): If checked, an email will be sent if a job has any exception (including non-fatal exceptions where CopyStorm/Medic recovered).
| | Notify Current User | No | If checked, the Salesforce User specified on the Main tab of CopyStorm/Medic will be notified in addition to the specified recipients (Send Email To). | ## SMTP Notifications ![](/files/HHM7wmMLmLnLhzVRTsuX) | Parameter Name | Required | Description | | -------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Email Host | Yes | The SMTP host used by your organization (used to send email). If you do not know this information ask your email administrator. | | Email From | Yes | The email address that will appear in the “From” field in sent notifications. Note that many email servers allow this value to be a fake email address (e.g. “”). | | Send Email To | Yes | Notification emails will be sent to this white space separated list of email addresses. | | Subject | Yes |

The subject line used for notification emails. Though it can be any text you want, there are a few special text values that will be replaced dynamically before a message is sent:

  • “{status}” will be replaced by “OK” or “ERROR” based on the outcome of the job.

    • If there are any exceptions encountered then “ERROR” will be substituted, otherwise “OK” will be used.
  • “{application}” will be replaced by “CopyStorm/Medic”.

  • “{file}” will be replaced by the base name of the CopyStorm/Medic configuration file used by the job.

    • For example, the configuration file “C:\capstorm\config\myCopyStormMedicConfig.copyStormMedic” will be replaced with “myCopyStormMedicConfig.copyStormMedic”.
| | Send Notification On | No |

Determines when CopyStorm/Medic will send a notification email:

  • Success: If checked, an email will be sent if a job works perfectly (no exceptions).
  • Exception(s): If checked, an email will be sent if a job has any exception (including non-fatal exceptions where CopyStorm/Medic recovered).
| | Require Encryption | No | If checked, CopyStorm/Medic will require that connections to the email server be encrypted with TLS. | | Email User | No | Username used to authenticate with the SMTP server. | | Email Password | No | Password used to authenticate with the SMTP server. | # Global Parameters The CopyStorm/Medic Global Parameters page sets parameters that apply to all CopyStorm/Medic jobs. The Global Parameters page can be found by selecting the “Other Tools” dropdown in CopyStorm/Medic, and clicking “Global Parameters” in the dropdown list. ![](/files/ERqKmTjtbfj59nEmeEam) | Parameter | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | API Threshold | Do not run CopyStorm/Medic jobs if Salesforce API usage is above the provided percentage when the job is starting. For example, if you enter “50” in this input then CopyStorm/Medic will not run jobs if Salesforce API usage is above 50% of the maximum. | # Fix IsDeleted The CopyStorm/Medic Fix IsDeleted tool identifies records that have been deleted in Salesforce, removed from the Salesforce recycle bin, but not marked as deleted in CopyStorm. This situation occurs when: * A user deletes records in Salesforce and empties the recycle bin BEFORE CopyStorm has run. * A system administrator empties the global recycle bin. * A Salesforce API program does a hard delete on Salesforce records. A hard delete bypasses the Salesforce recycle bin mechanism. Under normal circumstances, when a user deletes a record in Salesforce the record is not really deleted immediately. Instead, it is placed in the Salesforce recycle bin. The mechanism used by Salesforce is a hidden field named “isDeleted”. When CopyStorm runs it copies all modified records — including those where the “isDeleted” flag is set to true. After records are copied by CopyStorm, they are optionally deleted in the CopyStorm database based on parameters set on the Advanced tab. If normal Salesforce accounts are not permitted to empty their recycle bin (a good practice) then the condition that needs this tool will likely not occur. ## Running Fix IsDeleted in a GUI The UI for the Fix IsDeleted tool contains three parts: * The side of the dialog is used to select the tables to scan. * The top part of the dialog has parameters that determine if problem records should be fixed. * The lower part of the dialog displays the progress of a scan as it happens. ![](/files/G2KPhCvQ3v0rPh6xPXXj) ## Running Fix IsDeleted as a Batch Job The Fix IsDeleted tool can be run as a batch job using the following syntax: * Windows * CopyStormMedic.bat -tool fixDeleted -run configFile.copyStormMedic * Linux * sh CopyStormMedic.sh -tool fixDeleted -run configFile.copyStormMedic ## Viewing Identified Problem Records By checking the “Show Problems” checkbox, the application will store a list of all record IDs that are identified as being nonexistent (deleted) in Salesforce. To view the record IDs, run the tool, and after it is completed click the “Show Records” button. This will open up a modal window displaying the Table name, and Salesforce Record ID of all the records that were identified by the tool. The “Show Problems” setting can be used to validate the data prior to anything being deleted. To use it in this way, ensure that “Show Problems” is enabled, and “Fix Problems” is disabled. With this configuration, the tool will search for and identify problem records, and generate a list of IDs that can then be used to query the backup database to validate data prior to deletion, or for use in identifying mistakenly deleted records in Salesforce. In the Record viewer window, the resulting list of TableName:RecordID pairs can be exported as a .csv as well, using the “Export” button at the bottom of the modal window. See below for an example of the record viewer window. ![](/files/A8aOHXBxNN9DNXXGQYSl) ## Viewing Identified Problem Records – When Running Via Terminal When running the tool via command line, and using a config file which has the “Show Problems” flag enabled, the command line output of the application will include a list of all identified problem records. An example of this is shown below. ![](/files/C9IuwXFcntS2YeAPi8cW) # Refresh Columns The CopyStorm/Medic Refresh Columns tool replaces the value of one or more columns in CopyStorm with their current values in Salesforce. This tool is typically used when the definition of a formula is modified in Salesforce or when a previously hidden column becomes visible to the user CopyStorm uses to authenticate with Salesforce. When one or more columns in CopyStorm do not match the current value in Salesforce there are two methods which can fix the problem: * Perform a forced update on the CopyStorm database using CopyStorm. This technique works but will cause every column in the CopyStorm tables to be read from Salesforce and written to CopyStorm, including columns whose data is already synchronized. * Use this tool to limit reads/updates from Salesforce to specific columns. This technique can be many times faster than a forced update, and does not re-synchronize data that is already synchronized. ## Running Refresh Columns in a GUI To access the Refresh Columns GUI: * Select “Other Tools” from the top-level menu. * Select “Data Synchronization”, then “Refresh Columns”. The UI for the Refresh Columns tool contains three parts: * The side of the dialog is used to select the tables and columns to scan. * Click on a table name to launch a pop-up dialog for column name selection. * The top part of the dialog has parameters that determine the date range of records to be updated. * Leave these parameters blank to scan all records. * The lower part of the dialog displays the progress of a scan as it happens. ![](/files/6WHRXrizlSvYdSxYVs4T) Clicking on a column name will cause a column selection dialog to appear: ![](/files/gLdHfPUwEU3ZhU2OQJes) ## Managing Stale Formulas Automatically When a formula column definition changes in Salesforce, timestamps on Salesforce records do not change. In practice this means that a formula column in the CopyStorm database will not reflect the proper value after a formula definition change until the next time a non-formula field on the record changes. Starting with CopyStorm 7.34.10, both formula column definition changes and the last formula column refresh time are tracked. The “Manage Stale Formulas Automatically” option takes advantage of this tracking and, when checked, will: * Automatically identify all formula columns whose definition has changed since their last refresh in CopyStorm, and add them to the column refresh set. * Once the formula columns are refreshed, CopyStorm will update the formula refresh time in its tracking tables. ## Running Refresh Columns as a Batch Job The Refresh Columns tool can be run as a batch job using the following syntax: * Windows * CopyStormMedic.bat -tool refreshColumns -run configFile.copyStormMedic * Linux * sh CopyStormMedic.sh -tool refreshColumns -run configFile.copyStormMedic # Fix ContentDocumentLinks The *Fix DocLinks* tool repairs records in the Salesforce ContentDocumentLink table. The Salesforce ContentDocumentLink table has query restrictions that make it impossible to efficiently perform an 100% accurate incremental backup. There are two primary reasons that block an 100% accurate backup. 1. The ContentDocumentLink table cannot be queried only on its SystemModStamp field. 2. ContentDocumentLink permission values can be changed and there is no way to find the changes efficiently because of reason #1. In practice most customers will either never run this tool or run it on a weekly basis. ## Running Fix DocLinks in a GUI To access the Fix DocLinks GUI: * Select “Other Tools” from the top-level menu. * Select “Data Synchronization”, then “Fix DocLinks”. The UI for the Fix DocLinks tool is minimal. * The top of the dialog controls how many concurrent queries to run against Salesforce and what data ranges to scan. * The middle part of the dialog has a large button for starting the search and text for reporting progress ![](/files/SBufnWzljMmqjUqBv2FH) The tool will report progress as it scans records and when finished will look like: ![](/files/SQsTVjvuvsOx9ROhN5lG) where: * Candidates = total # of ContentDocumentLink records scanned * Inserted = number of ContentDocumentLink records inserted into the CopyStorm database. * Update = number of ContentDocumentLink records updated in the CopyStorm database. ## Running Fix DocLinks as a Batch Job The Fix DocLinks tool can be run as a batch job using the following syntax: * Windows * CopyStormMedic.bat -tool fixDocLinks -run configFile.copyStormMedic * Linux or Mac * sh CopyStormMedic.sh -tool fixDocLinks -run configFile.copyStormMedic # Refresh Formulas The CopyStorm/Medic Refresh Formulas tool replaces the values of formula columns in CopyStorm with their current value in Salesforce. This tool is typically used when the definition of a formula is modified in Salesforce. CopyStorm minimizes the amount of records copied from Salesforce by only copying records whose modification date has changed. When a formula column is modified in Salesforce record modification times do not change so CopyStorm does not pick up the new formula values for older records. There are two ways to update this data in the CopyStorm database: * Performing a forced update on the CopyStorm database using CopyStorm. This technique works but will cause every column in the CopyStorm tables to be read from Salesforce and written to CopyStorm. * Using this tool to limit reads and updates from Salesforce to formula columns. This technique can be many times faster than a forced update. ## Running Refresh Formulas in a GUI To access the Refresh Formulas GUI: * Select “Other Tools” from the top-level menu. * Select “Data Synchronization”, then “Refresh Formulas”. The Refresh Formulas tool contains three parts: * The side of the dialog is used to select the tables to scan. Note that if a table without formulas is selected it will be silently ignored by the tool. * The top part of the dialog is used to specify the date range of records to be updated. Leave these parameters blank to scan all records. * The lower part of the dialog displays the progress of a scan as it happens. ![](/files/445nTfJYNKfArHBJArNt) ## Running Refresh Formulas as a Batch Job The Refresh Formulas tool can be run as a batch job using the following syntax: * Windows * `CopyStormMedic.bat -tool refreshFormulas -run configFile.copyStormMedic` * Linux * `sh CopyStormMedic.sh -tool refreshFormulas -run configFile.copyStormMedic` # Scan Timestamps The CopyStorm/Medic Scan Timestamps tool performs an exhaustive comparison of Salesforce versus CopyStorm record modifications timestamps. This tool is provided for two reasons: * It can be used to provide assurance that CopyStorm is copying all records properly. * If Salesforce has an “after update” trigger on a table where the trigger updates the source table, this tool can be used to help track down the problem trigger. Comparing modification timestamps from Salesforce with CopyStorm is a trickier task than one might think — here are a few of the reasons why: * Salesforce stores timestamps to the nearest millisecond. * Microsoft SQL/Server stores timestamps to the nearest 3 milliseconds. * MySQL stores timestamps to the nearest second. * Database to GMT time conversion works differently on different databases. The Scan Timestamps tool will report the table, record id, timestamps, and timestamp difference for each record in Salesforce whose timestamp does not match CopyStorm and for records that are missing from CopyStorm. Records may be missing from CopyStorm due to being missed in a sync or because they will be synced in the future. ## Run Scan Timestamps in a GUI To access the Scan Timestamps GUI: * Select “Other Tools” from the top-level menu. * Select “Database Validation”, then “Scan Timestamps”. The UI for the Scan Timestamps tool contains four parts: * The side of the dialog is used to select the tables to scan. * The top part of the dialog has parameters that determine the date range of records to be updated. Leave these parameters blank to scan all records. * The lower part of the dialog displays the progress of a scan as it happens. * If an error is detected a button will appear in the lower right hand corner. Click on the button to see error details and/or export the errors to a file. NOTE: We purposefully corrupted the CopyStorm database so the “Show Problems” button would appear in the following screen shot. ![](/files/TzgHrRagnY14ebZ3lKII) If the Show Problems button appears, it can be clicked to show the details for each problem record. In this example we corrupted the CopyStorm database by hand so the Salesforce database would contain records missing from CopyStorm. The following screen shot is the result of clicking on the “Show Problems” button: ![](/files/kQ5sAlyMgsdO0oPt9KlY) ## Running Scan Timestamps as a Batch Job The Scan Timestamps tool can be run as a batch job using the following syntax: * Windows * `CopyStormMedic.bat -tool scanTimestamps -run configFile.copyStormMedic` * Linux * `sh CopyStormMedic.sh -tool scanTimestamps -run configFile.copyStormMedic` # Table Editor The CopyStorm/Medic Table Editor tool is for non-DBA people who want to remove data or tables from a CopyStorm database. Why would someone want to do this? * You may want to drop a specific table so CopyStorm will rebuild it the next time it runs. This case comes up when a column definition in Salesforce has changed and the corresponding change is not supported by the target database. * You may want to drop a Salesforce Feed table, like AccountFeed, so CopyStorm will replace it with a database view the next time it runs. * You may want to make CopyStorm completely forget about a database table it created. This action drops a table and all references to it in the CopyStorm metadata. Without the Table Editor tool a DBA can always manually truncate or drop a table — this tool is designed to provide a way for a non-DBA to perform the same task. ## Running Table Editor in a GUI To access the Table Editor GUI: * Select “Other Tools” from the top-level menu. * Select “Rarely Used”, then “Table Editor”. The UI for the Table Editor tool contains three parts: * The side of the dialog is used to select the tables to truncate, delete, or forget.. * The top part of the dialog determines if the selected tables will be deleted, truncated, or forgotten. * The lower part of the dialog displays the progress of table modifications. ![](/files/XRjjzyHZGedCW0H4Rrhx) ## Running Table Editor as a Batch Job The Table Editor tool cannot be run as a batch job. # Find Dirt ## Find Dirt The CopyStorm/Medic Find Dirt tool finds data in Salesforce fields that should not be possible. Capstorm provides this tool because, as a Salesforce instance ages, it will often accumulate data that does not match the field definition. Here are a few examples: * A string field is limited to 80 characters but has a few records where 96 characters are stored. * A numeric field is limited to zero digits after the decimal point but Apex code inserted records where digits appear. This tool will report the table, record id, column name, and problem description for any field that does not match the Salesforce field definition. ## Running Find Dirt in a GUI To access the Find Dirt GUI: * Select “Other Tools” from the top-level menu. * Select “Salesforce Data Quality”, then “Find Dirt”. The UI for the Find Dirt tool contains four parts: * The side of the dialog is used to select the tables to scan. * The top part of the dialog has parameters that determine the date range of records to be scanned. * Leave these blank to scan all records. * The lower part of the dialog displays the progress of a scan as it happens. * If an error is detected a “Show Problems” button will appear in the lower right hand corner. * Click the button to see error details and/or export the errors to a file. If the “Show Problems” button appears, click it to show the details for each problem record. ![](/files/nfOiVslRFek2fWYQ6YN6) ## Running Find Dirt as a Batch Job The Find Dirt tool can be run as a batch job using the following syntax: * Windows * `CopyStormMedic.bat -tool findDirt -run configFile.copyStormMedic` * Linux * `sh CopyStormMedic.sh -tool findDirt -run configFile.copyStormMedic` # Permission Checker The CopyStorm/Medic Permission Checker will find Salesforce tables and fields that are visible using the Salesforce metadata API but are not visible for backup using the current Salesforce credentials. It is a common misconception that a member of the Salesforce Administrator group can see all table and fields in a Salesforce instance. In practice, there are even many standard Salesforce fields that a Salesforce Administrator cannot view by default. The Permission Checker helps you find these tables and fields. ## Running the Permission Checker in a GUI Running the Permission Checker involves clicking on a single button – Start Scanning Metadata. * The Salesforce user credentials provided to the Permissions Checker must have access to read the Salesforce metadata definitions for each object. * Use of the metadata API requires the system permission “Modify Metadata Through Metadata API Functions.” ![](/files/NnJbUUIUzL0V4V5Oxgie) Do not be alarmed when you find a lot of fields that are not visible to a Salesforce Administrator. Most likely you are not using them in your instance. ## Running Permission Checker as a Batch Job The Permission Checker tool can be run as a batch job using the following syntax: * Windows * CopyStormMedic.bat -tool findPermissionErrors -run configFile.copyStormMedic * Linux * sh CopyStormMedic.sh -tool findPermissionErrors -run configFile.copyStormMedic # Base64 Export When storing Files and Attachments in the CopyStorm database, CopyStorm mirrors the Salesforce API specification for the field. This means that Salesforce object fields of type Base64 (such as Attachment.Body and ContentVersion.VersionData fields) are stored in the CopyStorm database as Base64 text strings. Such representations are industry standard and are intended to eliminate problems that can arise with storing binary values in a database. CopyStorm also removes limitations on the size of large text fields by means of a process of “chunking” – breaking down large blocks of content into chunks that fit comfortably in database given the unique datamodel limitations of the database engine. For example – Salesforce supports ContentVersion files as large as 2 GB, but the largest text string that can be stored in a PostgreSQL database is approximately 1 GB. With the “chunking” system, there are no practical limitations to how large a field value can be stored. This storage format can make it complex to view the contents of a Salesforce File or Attachment. For example, if the Base64 content is a PDF file, then you cannot view the PDF using a normal PDF viewer without exporting the Base64 text from the database and decoding the file content. The Base64 Export tool provides a way to directly export Files and Attachments from the CopyStorm database. This tool is also compatible with all database storage types used by CopyStorm – Base64 strings, chunked files, and CS:Govern-encrypted files. Just as there are no practical limits in how large a document can be stored in the primary CopyStorm database or a Snapshot, there are also no practical limits on how large a document can be exported to the file system (as long as you have enough disk to store the file!). ## Running Base64 Export in a GUI To run the Base64 Export tool in the CopyStorm/Medic GUI, open the tool from the main menu using Other Tools => Data Synchronization => Export Base64 Objects to File System. This will open the following GUI: ![](/files/v7zok604nneztAMZb4l5) The DataSet Selector drop-down box in the top-left of this tool contains a listing that includes the primary CopyStorm Database along with any defined Snapshot policies that the user has defined. In the diagram above, the primary CopyStorm Database has been selected which means that the Base64 Exporter will export files stored in the main CopyStorm data tables. Below that drop-down box is the list of Available Table Fields. The Available Table Fields list allows selection of the specific File or Attachment object and field(s) to be exported. ## Base64 Export Parameters When exporting content out of the source DataSet and into the file system, the tool, minimally, must know where to place those exported files. CopyStorm/Medic creates the output location by combining the “Output Path” folder path and a rule that specifies which sub-folders, if any, the files should be exported into. The Output Path setting is shown in the GUI screen shot above, and the sub-folder rule is provided by the Folder Names drop-down box. This drop-down has two choices: * **No Sub-Folders**: Files are exported into a flat structure in the Output Path folder location. * **ObjectType**: Files are exported into a folder whose name is based upon the Salesforce object name. For example, all ContentVersion documents will be exported into a folder named \[Output Path]/ContentVersion/. As not all File objects in Salesforce provide a file name field, the Default File Extension is used to provide a reasonable default value to use in cases where a file name cannot be established. The Decrypt Compliance Data checkbox is used to tell the Base64 Export tool whether it should attempt to decrypt a Base64 field that has been encrypted with CS:Govern. If the field is encrypted with CS:Govern, then this checkbox will cause CopyStorm/Medic to attempt to decrypt the field values – if you have permission to view the plaintext for the fields, then it will export the file data. If you do not have permission to view the plaintext, CopyStorm/Medic will export the CS:Govern masked value. ### File Rules To configure rules for exporting files from a specific field, click on the field name in the left-hand side of the Base64 Exporter. This will display a dialog that looks like the following: ![](/files/i1bj5Q0lMYcQws3TRJyB) This dialog allows configuration of a number of parameters, selecting which records should be exported along with setting restrictions to limit the total disk consumption used by the exporter. The Base64 Export tool allows limits on both the number of files to be exported as well as the total disk space taken up by the export process. It is important to know that these limits are per-object based. For example, if you enter 5 into the Max Exported Files field for the Attachment object along with a value of 15 for the ContentVersion object, and there are more than five Attachment and more than 15 ContentVersion records that match the selection parameters, then 20 total files will be written. There will be 5 Attachment files written to the disk, and 15 ContentVersion records similarly written. A blank value for the Max Exported Files sets there to be no limit – ALL scanned candidate records will be exported for all objects and records matching the selection criteria. This is also true for the Max Disk Space Consumption parameter – if this parameter is set to blank, then CopyStorm/Medic will not limit the amount of disk used by the export. Having values entered for BOTH the Max Exported Files as well as the Max Disk Space Consumption data entry fields causes both rules to be applied – whichever limit is exceeded first will stop the processing of the object type. Having limits on the Max Exported Files as well as the Max Disk Space Consumption helps you prevent runaway situations where millions of files or terabytes of data match the criteria – filling up all space on the output drive. You can also play with the tool with a small number for one or the other (or both) and, once comfortable with how it works, leave those data entry fields blank to allow for a complete export. ### File Name and Duplicate (Clobbering) Rules The Base64 Export tool will use the following logic to determine the name of the file to be exported to the file system: 1. First the tool will look for a title field in the scanned candidate record. If the title field is not blank, then the title field value will be used. If there is no title field or the title field is blank, the SalesforceId will be used as the file name. 2. If the selection for the File Names drop-down box is PreserveFileName then we are done and we know the name of the file to be exported – it is the one from step (1). 3. If the selection for the File Names drop-down box is ID\_Filename and a title field value was found in step (1), the file name will be the Salesforce record Id followed by an underscore and followed by the title found in step (1). Otherwise the filename is just the Salesforce Id. 4. If the selection for the File Names drop-down box is Object\_ID\_Filename and a non-blank title was found in step (1) then the file name will be the Salesforce record’s Object name followed by an underscore and then followed by the Salesforce record Id followed by an underscore and followed by the title found in step (1). Otherwise it will be the Salesforce record Object name followed by an underscore and then followed by the Salesforce record Id. If exporting files to a directory that already has files in it, then CopyStorm/Medic may create new output files and not overwrite existing files, or CopyStorm/Medic may overwrite the existing files with a new export. These rules also apply to exports that may include duplicate file names – for example: * Files were exported to the target folder from an earlier CopyStorm/Medic session. * A target Output Path was selected which just happened to already contain some files some of which some just happened to have the same name as files to be exported. * The user clicks the Export button multiple times – each click causing the tool to re-scan and re-export candidate records. * The user has selected the No Sub-Folders option for the Folder Names drop-down box and there happen to be similarly titled documents in Salesforce even though the Object types are different. For example, there might be an Attachment called “Company Regulations.pdf” as well as a ContentVersion object with that exact same name. The Duplicate Rule drop-down box is used to configure the Base64 Export tool to handle these situations. Here are the options: * **Skip**: If the tool attempts to write a file to the file system that already exists in the target folder, it will simply skip exporting that file (in other words it will **not** clobber the existing file.) * **Overwrite**: If the tool attempts to write a file to the file system that already exists in the target folder, it will simply overwrite that file (in other words it **will** clobber the existing file.) * **Version**: If the tool attempts to write a file to the file system that already exists in the target folder it will suffix the new file with the conventional syntax for such cases – a space followed by parentheses that enclose a monotonically increasing number. Behind the scenes the tool will continuously bump that number until a unique file name is generated. ### Record Selector Rules Next to the File Rules tab in the screen shot above of the Base64 Export Rules dialog, there is a tab called *Select Records*. Clicking on that tab will show a view that looks like this: ![](/files/8SS9eNn0F5zMlkAuUz6m) Like the File Rules, the Select Records tab defines rules that are specific to the selected Object from the Available Table Fields list. These rules, however, have nothing to do with how and where files are stored in the target file system. Instead they are used to determine which records in the selected DataSet are candidates for export. This is a much more strategic and surgical rule than the file rules, and the functionality provides similar options as in the CopyStorm/Restore application. For details, see the [Records to Restore Editor](https://learn.capstorm.com/copystorm-restore/readme/reference/restore-set-editor-tab/records-to-restore-editor) documentation in the CopyStorm/Restore section. Note that not all of the UI elements used in the dialogs for Restore apply to Base64 Export. ### Point-In-Time Export From Snapshot Beyond selecting which records should be exported, the Base64 Exporter also allows point-in-time selection of matching records. The following screen shot shows available options for Point-In-Time Export. These are available when a Snapshot is selected in the DataSet Selector: ![](/files/RecQm2DdEic6pvngdqzP) When selecting a Snapshot from the DataSet Selector, you have options for determining which points in time the Base64 field values should be exported from. The screen shot above is showing the *A Point In Time* selection. Here are the available choices: * Most Recent * A Point In Time * A Single Date * Between Two Times An in-depth discussion of how the CapStorm Snapshot sub-system works can be found [here (How do CopyStorm Archives Work)](/frequently-asked-questions/how-do-copystorm-archives-work). The following shows available options when the Most Recent selection is made: ![](/files/caigQ5mDQNLjHiGUWys4) The following screen shot shows available options when the *A Single Date* selection is made: ![](/files/6pT52wHvDh1PwqIb1DKd) The following screen shot shows available options when the *Between Two Times* selection is made: ![](/files/S2mCYFKcWrJ77hJAHMUA) If either of the following options are selected, it is possible that multiple files match the selection criteria: * **A Single Date**: The time span starts at midnight of the selected date and ends just prior to midnight of the following day – resulting in a 24 hour period. * **Between Two Times**: The time span starts at the specified start time and ends at the specified end time. In those two cases it is very possible that there are many records for the selected Salesforce object of a specified Salesforce Id. To choose which version(s) of the record to export, you can select the Earliest, the Latest, or All, versions of that Salesforce Id. ## Adding a Field to the Base64 Exporter Available tables are provided by a *Base64ExportTableRuleRegistry.xml* configuration file. This file contains metadata that defines which Salesforce object fields are candidates for the Base64 Export process, and contains the following built-in definitions. To add a new table to the Base64 Exporter, create a new *Base64ExportTableRuleRegistry.xml* file and add it to the \[CopyStormMedic]/config/ directory. ![](/files/logyqQVFW366y9d7hinf) Each rule in this XML document specifies the following: * The name of the Salesforce object (*tableName*) * The name of the Salesforce Base64 field (*contentsFieldName*) * The name of the Salesforce field whose value is the title of the document (*titleFieldName*) * The name of the Salesforce field whose value is the length (in bytes) of the document (*contentLengthFieldName*) All attributes are required with the exception of the *titleFieldName* attribute. If the *titleFieldName* attribute is not provided, the Salesforce Id field value will be used to name the exported files. ## Running Base64 Export as a Batch Job The Base64 Export tool cannot be executed as a batch job. # Base64 Sanity Checker When the Salesforce REST API returns a record with a BASE64 column, the BASE64 data is returned as a hyperlink that can be used to read the raw bytes. Experience has shown that it is possible for the hyperlink returned by Salesforce to return a HTTP 404 (not found) or an HTTP 500 (internal lightning platform error) rather than the expected HTTP 200 (OK). When this happens: * The BASE64 data cannot be read by the REST API. * The same operation using the SOAP API results in an UNKNOWN\_EXCEPTION or an unparsable SOAP body The Base64 Sanity Checker tool tests the hyperlinks returned by the REST API and reports the ID of records where the hyperlink fails (e.g. Does not return a HTTP 200). It is often wise to check each reported record using the Salesforce GUI to make sure that the data is really available. If you find a corrupted Attachment record report it to Salesforce. ## Running The Base64 Sanity Checker in a GUI To access the Base64 Sanity Checker GUI: * Select “Other Tools” from the top-level menu. * Select “Salesforce Data Quality”, then “Base64 Sanity Checker”. ### Step 1: Set Search Parameters The only required parameters are the tables to search, but there are a number of other parameters you may want to modify. | Parameter | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Modified After | Search for records starting at this date. A date can be selected from the menu or typed in directly. Values are restricted to datetime formats supported by SOQL. | | Modified Before | Search for records up to this date. A date can be selected from the menu or typed in directly. Values are restricted to datetime formats supported by SOQL. | | Salesforce Batch Size | Number of records to process from Salesforce in a single batch. The default usually provides optimal performance. | | # Checker Threads | Number of base 64 hyperlinks to analyze concurrently. | | Max Timeout (sec) | Number seconds to allow for each hyperlink test. If a hyperlink HTTP request times out it is assumed to be OK based on Capstorm’s experience with problem records. Setting this to a smaller value may improve the overall throughput of the analysis. | | Report Timeouts | If this option is checked then all HTTP timeouts will be reported as potential problems. Enabling this feature may be useful in selecting an optimal *Max Timeout* value for your organization. | ![](/files/wPoixBvTD0kGYNVe5MCh) ### Step 2: Check Hyperlinks The next step is to click on the *Start Sanity Check* button and wait for the analysis to complete. Note: * Each hyperlink check takes one Salesforce API call. There is one additional API call for every 2000 records. * An running analysis can be cancelled but the cancellation process may not be immediate. The reason is that pending HTTP requests must complete or timeout before the cancellation process is finished. ## Summary Hopefully when you scan your Salesforce the tool will not find potential problems. However, if potential problems are found then it is wise to check the validity of each record using the Salesforce GUI. If you are using CopyStorm to perform backups, then this class of error will be discovered during the normal course of a backup. Unfortunately the error is often masked by an UNKNOWN\_EXCEPTION or an Invalid SOAP Envelope response from Salesforce. # Delete Salesforce Records To access the Delete Salesforce Records tool: * Select “Other Tools” from the top-level menu. * Select “Rarely Used” and then “Delete Salesforce Records” ![](/files/GozUgBKw8OY5CzaKZkrC) Understanding the consequences of deleting a hierarchy of records and even performing a hierarchical delete are difficult problems. Examples: * What records will be deleted in Salesforce as a side effect if I delete the ACME account? * Delete the following set of old Emails AND all of their related records. The Delete Salesforce Records tool minimizes the effort required to understand, generate audit trails, and execute hierarchical type deletions. The general flow of the Delete Salesforce Records tool is fairly simple. * Select a Salesforce object as the root object for deletion (e.g. Account.) * Use a record selector to identify top level records of this root object to delete. * Optionally explore the records that will be deleted. * Delete the selected records and all children using one of the following techniques: * Generate Data Loader files which can be use to manually delete records. * Generate a Python script which, when run, will delete the records. * Delete the records immediately using API calls to Salesforce. The following sample screen shot illustrates that the *Account* Salesforce object is being selected. ![](/files/O44rcJAiMLYusdPh14Vg) Once you have selected your Salesforce root object, you will be prompted to allow loading the relationship data for that object. This process may take a few seconds to complete. Upon completion you will be presented with a screen that shows these relationship details and the Select Records wizard tab at the top will be enabled. ![](/files/IJhQTDQkYI874wVSxW1O) ## Dependency Plans In the screen shot above the left side of the relationships panel is a tree structure that represents the hierarchy of Salesforce relationship (or dependency) objects on the Salesforce root object that was selected earlier (Account in this example.) Throughout this document we will be making reference to this *Dependency Plan*, and in the current implementation of the Delete Salesforce Records tool, there is only one root Salesforce object for that plan. The terms *relationship* and *dependency* are interchangeable. In this case the dependency plan is used to define rules for Salesforce record deletion. As one selects objects from the left-side dependency hierarchy tree, the dependency objects on that selection will appear on the right side along with some important metadata, as shown in the screen shot below. ![](/files/GvoL395vveyD48T0JBaT) There are various ways in which rules can be applied surgically on each relationship object. One quick means is to right click on a relationship on the left side hierarchy tree and observe that a pop-up menu will be presented to the user, allowing for adding an exclusion rule, and possibly overriding the Cascade delete setting, as depicted in the screen shot below: ![](/files/G5VeJM7JXgjza2hgcWOH) ## Selecting Records to Delete Once a specific Salesforce object has been selected, the Salesforce Ids of the top level records to be deleted must be specified. This is done using a Record Selector. There are four ways to select top level record ids. ### **Snapshot** The Snapshot record selector uses a previously created CopyStorm Snapshot to determine which records are candidates for deletion. When this option is selected the only deletion candidates that can occur are those that are both in the specified snapshot AND are in the selected Salesforce object table. ### **SOQL** The SOQL record selector expects a SOQL statement which returns top level records Ids. For example, if the selected Salesforce root object is the Account object then any of the following SOQL statements would work. * SELECT Id FROM Account WHERE name=’ACME’ * SELECT AccountId FROM Opportunity WHERE ShippingCountry=’Canada’ ### **Id List** The Id List record selector expects a text block containing top level set of record ids. A space or comma can be used to separate record ids. This is a useful selector when only a few top level records need to be deleted. ### **File** The File record selector expects a text file with one Salesforce Id per line. Once a Record Selector has been entered, click on the *Scan for Records* button to find all related records. The following example screenshot used a SOQL record selector to specify which records will be deleted. ![](/files/auWGDMStkmDnvCvkUOnF) ## Understanding Deletion Candidates Though it is possible at this point to start deleting records, it is often advisable to inspect the deletion candidates first. There are two clean ways to visualize what records will be deleted. * Clicking on a candidate table will cause its related deletion candidate records to be display on the right side of the display. * Generate a HTML report which show details about each record that will be deleted. This report can be useful as as audit trail. Clicking on a candidate table causes a table with the following columns to be displayed. * Salesforce Id — the Salesforce Id of the record to be deleted. * Reference Field(s) — the name of fields in the record which point to other tables also in the deletion set. * Parent Table(s) — the name of each table the reference fields in the record are pointing to. In the following screen shot, the first Asset record in the list was included because: * Three of its reference fields point to other tables in the deletion set (AccountId, ContactId, and RootAssetId) * The tables the three reference fields are pointing to are Account, Contact, and Asset. ![](/files/Rj0ZYadVRjVJElgN3L8v) For a complete audit trail and dependency details, the *Export* button is a good option ## Exporting Records To export deletion candidates, click on the “Export” button. ![](/files/0fvMsob2tMJkAjtVimtE) ### Generating a HTML Report To export a single-file HTML Report containing all of the record Ids that are marked to be deleted, click on the Generate HTML Report button. The HTML report will include Record Ids and dependency information detailing why each individual record is included in the deletion plan. ### Generating Data Loader Files To generate Data Load files click on the Generate Data Loader button and enter a output directory when prompted. The system will generate a series of 1 column text files where: * The only column in each file is a Salesforce Record Id. * Each file will be prefixed with a four digit number which indicates the order in which it should processed by Salesforce. For example, the data loader files generated for the example is this document are: ``` # # Record Deletion Plan Account # Generated Mon Sep 18 10:48:07 CDT 2023 # Output Dir /home/greg/tmp/unittest # # The following files should be deleted from Salesforce in order. # 0001CaseComment.txt 0002OpportunityContactRole.txt 0003QuoteLineItem.txt 0004OpportunityLineItem.txt 0005CaseFeed.txt 0006OpportunityFeed.txt 0007Case.txt 0008FeedItem.txt 0009Task.txt 0010AccountRelationship__c.txt 0011Quote.txt 0012Opportunity.txt 0013Asset.txt 0014Contact.txt 0015Account.txt # # # End of Data Loader File Plan Account # ``` ### Generating Python To generate a Python script which uses the CapStorm SQLForce library to delete records, click on the Generate Python button and enter an output directory when prompted. The system will generate a main python script and a directory of data files. Running the generated script will delete records from Salesforce. ## Deleting Records The final step is to delete records from Salesforce using one of the following techniques. * Generate Data Loader files which can be use to manually delete records. * Generate a Python script which, when run, will delete the records. * Delete the records immediately using API calls to Salesforce. ### Deleting Records Immediately Clicking on the *Delete* button will prompt a user for confirmation and the rapidly start delete records from Salesforce. Before beginning the deletion, the confirmation dialog requires you to enter the Salesforce Org URL for the target Salesforce. * The Salesforce Org URL (which appears as a somewhat randomly generated phrase) must be typed in by hand. * If the deletion target is a production Salesforce instance then a checkbox must be selected. ![](/files/v5XiD44AMKD1DGZm0bgH)
ParameterDefaultMinMaxDescription
Records Per Batch2000150000The maximum number of records to process concurrently. Reducing this value lowers memory usage during very large deletions.
Records Per Salesforce Request2001200The maximum number of records in a single API call to Salesforce. 200 is both the default and the hard ceiling imposed by the Salesforce API - this value cannot be set higher.
Concurrent Salesforce Requests11100The maximum number of concurrent delete requests to send to Salesforce. Increasing this speeds up large deletions. Reduce this value if record locking errors are occurring frequently - a common symptom is retries appearing in the log.
Hard DeleteOn--When enabled, records are permanently deleted and will not be placed in the Salesforce recycle bin. This is the default behavior. Requires the "Bulk API Hard Delete" permission in Salesforce. Uncheck this if you want deleted records to be recoverable from the recycle bin.
Validate Before DeletionOff--When enabled, the system confirms each record still exists in Salesforce before issuing a delete request. Useful when time may have passed between scanning and deleting, and the record set may have changed.
Show Retry ReasonsOff--When enabled, if a deletion fails and the system retries, the retry reason is written to the log. The most common retry reason is record locking, which is typically resolved by reducing Concurrent Salesforce Requests.
Show Entity Delete ErrorsOff--When enabled, an error is displayed when attempting to delete a record that is already deleted. These errors are generally safe to ignore and are off by default.
Show Invalid Cross Reference ErrorOff--When enabled, an error is displayed for INVALID_CROSS_REFERENCE_KEY Salesforce errors. These are generally safe to ignore and are off by default.
![](/files/hlhC9yqkw1rsYdz8Cj5Q) ### Headless CLI The Delete Salesforce Records tool can be run in headless batch mode using the standard CopyStorm/Medic command line interface. This is useful for scheduling automated deletion jobs via Windows Task Scheduler or cron on Linux. To run in headless mode, first configure and test your deletion settings in the UI, then save a CopyStorm/Medic configuration file using File > Save As with the "Encrypted" and "Save Passwords" options selected. All deletion settings - including the dependency plan, record selector, and advanced options - are saved in this file. The tool name for the `-tool` switch is `deleteTool`. **Windows:** ``` CopyStormMedic.bat -tool deleteTool -run configFile.copyStormMedic ``` **Linux:** ``` sh CopyStormMedic.sh -tool deleteTool -run configFile.copyStormMedic ``` The `-tool` switch is case-insensitive - `deleteTool`, `deletetool`, and `DELETETOOL` are all valid. The following global CopyStorm/Medic switches can be used to override values stored in the configuration file at runtime: | Switch | Parameter | Description | | --------------- | ------------------------- | ----------------------------------------------------------------------- | | `-sftype` | `Sandbox` or `Production` | Override the Salesforce org type. | | `-sfuser` | Username | Override the Salesforce username. | | `-sfpassword` | Password | Override the Salesforce password. | | `-sftoken` | Security Token | Override the Salesforce security token. | | `-dbconnection` | Connection String | Override the database connection string. | | `-dbuser` | Username | Override the database username. | | `-dbpassword` | Password | Override the database password. | | `-dbschema` | Schema Name | Override the database schema. | | `-silent` | - | Suppress stdout progress output. Does not affect behavior or exit code. | There are no switches unique to `deleteTool` - all deletion configuration is controlled by the saved `.copyStormMedic` config file. The tool returns exit code `0` on success and `1` on failure. For the full global command line reference see CopyStorm/Medic. ### XML Dependency Plan Configuration The Delete Salesforce Records tool supports two user-provided XML override files that control which Salesforce objects are included in a deletion and in what order they are processed. These files do not replace the tool's built-in dependency logic - they are additive, meaning entries in your override files are merged on top of the built-in defaults. > **Important:** Because external files are additive only, you cannot remove a built-in exclusion via an override file. You can only add new entries. **File Locations** Place override files in one of the following directories. The tool checks these locations in order and loads any file it finds: | Priority | Path | | -------- | ------------------------------------- | | 1 | `${copystorm.root}/config/` | | 2 | `${capstorm.root}/config/` | | 3 | `${copystorm.config}/` | | 4 | `${capstorm.config}/` | These paths are controlled by Java system properties set by the product's startup scripts. If none of these properties are defined in your environment, no external override files will be discovered. *** **File 1: `SalesforceDependencyPlanOptions.xml`** This file controls which objects are excluded from deletion, which objects should have their Salesforce cascade-delete behavior overridden, forced deletion ordering between specific objects, and per-table performance tuning. The filename must be exactly `SalesforceDependencyPlanOptions.xml`. **Elements** **``** - Excludes an object from the dependency plan entirely. Use `name` for an exact match or `pattern` for a case-insensitive Java regex match. xml ```xml ``` **``** - Tells the tool to explicitly delete children of this object rather than relying on Salesforce's cascade-delete behavior. Use this when Salesforce's cascade-delete metadata for an object is unreliable. xml ```xml ``` **``** - Forces a hard deletion ordering between two objects. The `child` object will always be deleted before the `parent` object. Both attributes are required. This is a hard constraint - if the ordering cannot be satisfied, the tool will throw an error rather than violate it. xml ```xml ``` **``** - Sets per-table performance tuning values. Use the special name `__default__` to set fallback values for all tables that don't have their own rule. xml ```xml ``` | Attribute | Description | | ------------ | ----------------------------------------------------------------------------- | | `name` | Salesforce object API name, or `__default__` for the fallback rule. Required. | | `maxThreads` | Max concurrent Salesforce API threads for this object. Optional. | | `maxRequest` | Max records per Salesforce API request for this object. Optional. | | `maxBatch` | Max records to batch together. Optional. | **Full Example** xml ```xml ``` *** **File 2: `SalesforceDependencyOverride.xml`** This file handles cases where Salesforce's native relationship metadata does not correctly expose a parent-child relationship that the delete tool needs to follow. It lets you define a "parent as child" relationship so the tool discovers and deletes records it would otherwise miss. The filename must be exactly `SalesforceDependencyOverride.xml`. **Elements** **``** - Adds a synthetic child relationship to a given object. All four attributes are required - if any are missing the element is silently skipped. | Attribute | Description | | ------------------ | --------------------------------------------------------------------------- | | `tableName` | The object that gets the synthetic child relationship added to it. | | `referenceField` | The field on `tableName` that holds the foreign key pointing to the parent. | | `parentTable` | The parent object that will be treated as a child in the dependency tree. | | `relationshipName` | The name used to identify this relationship in the UI and dependency plan. | **Example** xml ```xml ``` *** **Gotchas** | | Detail | | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | **Additive only** | External files can only add entries - they cannot remove built-in exclusions or rules. | | **Exact filenames required** | The filenames `SalesforceDependencyPlanOptions.xml` and `SalesforceDependencyOverride.xml` are hardcoded. Any other name will be ignored. | | **`` is a hard constraint** | If the forced ordering cannot be satisfied, the tool throws an error rather than silently reordering. | | **All 4 `` attributes required** | If any attribute is blank or missing the element is silently skipped with no error. | # Delete Salesforce Records in Batch Mode The CopyStorm/Medic Delete Salesforce Records tool has the ability to run batch mode without the need of opening the application GUI and click through the steps of the tool. Execution of the tool in this manner requires a CopyStorm/Medic configuration file that already has all desired configuration options and filters selected and saved. For more information regarding the setup of this tool, see [here](https://learn.capstorm.com/reference/copystorm-medic/delete-salesforce-records). {% hint style="warning" %} When executing the Delete Salesforce Records tool through the CopyStorm/Medic GUI, there is a security pop-up that requires users to type the name of the target Salesforce org before beginning the deletion process. This security feature does NOT occur when executing in batch mode. {% endhint %} **Example Usage** This example is for Windows installations of CopyStorm/Medic. For Linux/Mac installations, change *CopyStormMedic.bat* to *CopyStormMedic.sh* ``` CopyStormMedic.bat -tool deleteTool -run "/path/to/config.copyStormMedic" ``` # Partition Editor The CopyStorm/Medic Partition Editor tool modifies the schema of the CopyStorm database to vertically partition CopyStorm Salesforce data tables into multiple database tables. This tool is typically used when a Salesforce object is wide enough that it cannot fit into a single SQL database table. For example, if the Salesforce Account and Contact objects both have the maximum number of custom fields **and** Person Accounts are enabled. This can cause a single Salesforce object to have more than 1,000 fields. In the scenario where partition creation fails for any reason, the next partition creation attempt will pick up where the last partition creation failed. When a partition is in a partial state due to a failure, CopyStorm jobs will skip over backing up records to this table until the partition is completed. This CopyStorm/Medic tool provides a way to resolve database error messages indicating that a record is too large to fit in a database table: * MySQL: “Row size too large (> 8126)” * SQL/Server: “Cannot Create a row of size 8765 which is greater than the allowable maximum row size of 8060” * PostgreSQL: “row is too big” ## Running the Partition Editor GUI To access the Partition Editor GUI: * Select “Other Tools” from the top-level menu. * Select “Partition Editor” in the dropdown. ![](/files/8KljA2sB5z6IdS2j8RQ8) A list of objects in the CopyStorm database is displayed on the left – select an object to view its partition configuration. After selecting an object, the following information will be displayed: 1. Partition Name 1. The CopyStorm database table name for the table storing the field. 2. Field Name 1. The Salesforce field name. 3. Column Name 1. The CopyStorm database column name for the column storing the field. 4. Type 1. The Salesforce field type. ![](/files/kwSKCvnU33PZdrp7Kt4Y) To move fields between partitions, or to create a new partition, click on the “Move Fields” button. This button will open a pop-up window displaying available partitions and configuration options to configure a new partition. The “Move \[Object] Fields To” drop-down controls the target partition. * If a pre-existing partition name is selected, then any fields already in the target partition will not be available to select. * If the “New JDBC Partition” option is selected, then CopyStorm/Medic will create a new database table to hold the selected fields. Configuration parameters for the target partition type will be displayed below the drop-down. Data partitions can be created in: * The SQL Database, as a new JDBC Table. * [A local folder on the CopyStorm server.](/reference/copystorm-medic/partition-editor/folder-partition-configuration) * [An AWS S3 Bucket.](/reference/copystorm-medic/partition-editor/s3-partition-configuration) The following screenshot displays the configuration panel for creating a new JDBC Database Table partition: ![](/files/egeQmyqoT0CXSrpIVLnH) Clicking on the “Continue” button will open a pop-up for Field Selection: * All selected fields will be moved to the target partition. * The source partition for each field is displayed next to the field name. ![](/files/8zxO1rDvKWGZOjIlyVX5) After clicking on the “Review Changes” button, the selected fields will be displayed – highlighting the source partition(s) that currently contain the fields. To perform the changes and update the CopyStorm database schema, click on the “Apply Changes” button. Clicking on the “Apply Changes” button will cause CopyStorm to: 1. DROP from the **target** partition any pre-existing columns for the selected fields. 1. This is to ensure that the partitioning process can be retried should any issues be encountered during the partitioning process. 2. Create schema in the **target** partition. 1. If “New Partition” was selected, a CREATE TABLE statement will be executed to create the new partition. 2. If a pre-existing partition was selected, an ALTER TABLE statement will be executed to add the selected fields to the target partition. 3. Copy field data from the **source** partition(s) to the **target** partition. 4. Update CopyStorm metadata for the selected fields. 5. DROP the data columns from the **source** partition(s). ![](/files/86jRdCW9dYR0lxBMYPkD) While the partitioning process is running, status will be displayed in the “Log” tab. When the partitioning process has completed, the log will indicate that the job has finished. ![](/files/pxBUveoU41Mc7EtoFFfL) After closing the pop-up, the Partition Editor will display the new status of the selected fields. ![](/files/avaTqmK5o0QXLvKBMFp1) # Folder Partition Configuration To configure a Folder partition in CopyStorm/Medic, provide the target folder: ![](/files/KdNFZezNZvtlguy9AIQf) To initialize the target directory, CopyStorm/Medic will: * Copy field data for the selected fields into the target folder. * DROP the database storage used by the selected fields. When CopyStorm replicates new or updated records, field values for the selected fields will be kept in files in the target directory. Record data is kept in nested folders generated using Salesforce Ids. # S3 Partition Configuration To configure a S3 partition in CopyStorm/Medic, provide connection information for the target S3 bucket. CopyStorm will use the AWS Default Credential Provider Chain to connect to S3. Environment variables, Java Properties, IAM Roles, etc… can be used to authorize CopyStorm for S3 access. ![](/files/vpiS66WpwGTWyYBkOsD7) To initialize the partition, CopyStorm/Medic will: * Copy field data for the selected fields into the target bucket, using the folder name as a prefix. * DROP the database storage used by the selected fields. When CopyStorm replicates new or updated records, field values for the selected fields will be kept in files in the target bucket. Record data is kept in a nested structure generated using Salesforce Ids. # How do CopyStorm Partitions Work? When replicating Salesforce data and schema to the CopyStorm database, CopyStorm will create database tables and columns which mirror the object and field structure in Salesforce. For example, there is a Salesforce object called “Account”, with a String field called “Name”. This means that the CopyStorm database will have a table called “Account” with a VARCHAR or CLOB field called “Name”. CopyStorm partitions use a technique called Vertical Partitioning to change the CopyStorm database schema by separating the database table for an object into multiple tables. Of course – CopyStorm, CopyStorm/Restore, and CopyStorm/Medic are fully compatible with CopyStorm partitions. For example, by default, the CopyStorm database may have an “Account” table with these columns: 1. Id 2. SystemModstamp 3. Name 4. BillingStreet 5. BillingCity 6. BillingState If the “Billing” fields are moved to a new partition, the database will have multiple tables: 1. The “Account” table, with these columns: 1. Id 2. SystemModstamp 3. Name 2. An “Account\_\_part1” table, with these columns: 1. Id 2. BillingStreet 3. BillingCity 4. BillingState Virtually any field can be moved to a partition if needed – with the exception of a small number of fields that require special handling. * The “Id” database column is always present on every partition. * A handful of fields are required to remain on the primary partition: * IsDeleted * Salesforce Modification Timestamps: * SystemModstamp * LastModifiedDate * Salesforce Audit Timestamps: * CreatedDate * LoginTime * SentDate * Fields that are encrypted with the CS:Govern application. Partitions may be created for virtually any object – the only exceptions are: * BigObjects * Tables that are included in the [CopyStorm Database View system](/frequently-asked-questions/how-does-copystorm-work/how-do-copystorm-views-work). # How are S3 Partitions Structured? When CopyStorm writes data to an S3 partition, each Salesforce record is stored as one or more files in your S3 bucket. The location of every file is derived entirely from the Salesforce record ID. There are no date folders, job ID folders, or table name folders. ## Path Format ``` s3://[bucket]/[folder]/[chars 0–4]/[chars 5–6]/[chars 7–8]/[chars 9–10]/[chars 11–12]/[filename] ``` The folder hierarchy is constructed by slicing the first 13 characters of the Salesforce record ID into five segments: | Segment | Characters | Example (ID: 001ZZ010000000000AA) | | ------- | --------------- | --------------------------------- | | Level 1 | 0–4 (5 chars) | 001ZZ | | Level 2 | 5–6 (2 chars) | 01 | | Level 3 | 7–8 (2 chars) | 00 | | Level 4 | 9–10 (2 chars) | 00 | | Level 5 | 11–12 (2 chars) | 00 | Full example path for the example ID: ``` s3://my-bucket/my-folder/001ZZ/01/00/00/00/001ZZ010000000000AA.xml ``` ## Files Per Record Each record produces at least one file and potentially more. **Main record file:** Always present, contains all standard field values as XML: ``` [recordId].xml ``` **Large field files**: Created for any field whose value exceeds 5 KB (e.g. `Body` on ContentVersion). The main XML file references these by name; the data is stored separately: ``` [recordId].[fieldName].txt ``` Example for a file attachment: ``` 001ZZ010000000000AA.xml 001ZZ010000000000AA.Body.txt ``` ## What Is and Isn't in the S3 Path | | Included in path? | | ---------------------------- | ------------------------------------------- | | Bucket name | Yes (configurable) | | Folder prefix | Yes (configurable, may be empty) | | AWS region | Configured separately, not part of the path | | Salesforce record ID | Yes. Determines all subfolders and filename | | Salesforce object/table name | No | | Org ID | No | | Backup date or job ID | No | # Help! Partition Creation Failed When partitioning tables containing large volumes of data or Base64-encoded content, the partition creation process can fail mid-way due to various issues, such as: * Application freezes or crashes * Transient network errors * Resource exhaustion (e.g. memory or CPU spikes) * Timeouts or I/O bottlenecks **These interruptions can leave partitions in an incomplete or inconsistent state.** The Partition Creation process is designed to be incremental: * Records are copied from Source to Target Partition in an ID-based ordering (lexicographical). * After each batch of records is copied from Source to Target, the last read Salesforce ID is persisted in the Database. * See the *CopyForceTablePartitionLoadTask* database table. **Note: CopyStorm will refuse to replicate data to a table if it has a Partition Creation In-Progress** **If partition creation fails, you can resume the process by navigating back to the Partition Editor Tool. You’ll see the following dialog appear if a partition creation is in-progress:** ![](/files/mzfkp29nraFqAQjV54Mi) From this point, you’ll have the option to: * Resume the Partition Creation process * This will start copying data from Source to Target from the Last Read Salesforce ID. * Cancel the Partition Creation process * **Note: Cancelling will not remove data already copied if the Source is S3 or a Folder** * Cancelling a JDBC Partition Creation will delete the new JDBC Partition. * **Source data will never be deleted during the cancellation process.** ![](/files/hpMRVp3nXeAMIApC7U6h) # Lineage Maintenance The CopyStorm/Medic Lineage Maintenance Tool is used to view lineage data & invalidate problematic lineage records. Over time, a Record’s lineage may be invalidated: * One of the legacy lineage entries may no longer exist in the CopyStorm Database or Snapshot. * Records may have multiple children, causing split lineage. For more details on Record Lineage, [refer to this article](https://learn.capstorm.com/copystorm-restore/readme/frequently-asked-questions/how-does-record-lineage-work). ## Using the Tool * Select “Other Tools” from the top-level menu. * Select “Data Synchronization” and then “Lineage Maintenance” ![](/files/1vt3JJi5vu1Px5HNznQH) * Optionally Configure Modified After & Modified Before * This will control what date ranges are queried for lineage. * Leaving these values Blank will query all time lineage. * Optionally Enable Validators * Purged Record Validation * Identifies lineage entries that do not have a corresponding physical record in the CopyStorm Database or Snapshot. * Branching Lineage Validation * Identifies lineage entries that have multiple children. * Optionally toggle “Only Show Invalid Lineages” checkbox * This should be used when investigating invalid lineages. * Click “Start Scan” ![](/files/ATscrmPCmmzUZOQ6qCIt) After clicking “Start Scan”, the application will render a view of: * All Lineage within the provided Modified After / Modified Before values. * The Status of each Lineage. * Valid * If there are no branches and all lineage entries exist in the CopyStorm Database or Snapshot * Invalid * If branch(es) are detected or any entry is missing from the CopyStorm Database or Snapshot * The associated Salesforce Table for the Lineage. * The Salesforce ID of the Lineage HEAD * HEAD is the most recent entry in the Lineage. Use the “Invalidate Selected Lineage” to mark problematic Lineage Entries as Invalid in the database. **Note: Invalidating Lineage will prevent CopyStorm/Restore from considering invalid entries as recovery candidates.** ![](/files/BkJzyVQGttnogtEoOI8r) **An example of viewing a full lineage is shown below:** ![](/files/82pPshff7jILcwYf2Zsu) # Count Records The CopyStorm/Medic Count Records tool performs a comparison of object record counts in Salesforce versus CopyStorm. This tool is provided as a sanity check for backup validation, to provide assurance that CopyStorm is copying all records properly. ## Run Count Records in a GUI To access the Count Records GUI: * Select “Other Tools” from the top-level menu. * Select “Database Validation”, then “Count Records” The UI for the Count Records tool contains four parts: * The left-hand side of the dialog is used to select the tables to scan. * The top part of the dialog has the following parameters: * of Tables to Process Concurrently. * Specifies how many tables the tool will process in parallel. Higher values may improve performance but increase resource usage. * Output To CSV *(checkbox).* * If checked, headless runs of Count Records will output tool results to STDOUT and Database Logs in CSV-compliant format. * The lower part of the dialog displays the progress of a record count analysis as it happens. * If an error is detected, a button will appear in the lower right hand corner. Click on the button to see error details and/or export the errors to a file. ![](/files/Xigb7x0L7rWc9LUXOsSz) ## Running Count Records as a Batch Job * Windows * `CopyStormMedic.bat -tool countRecords -run configFile.copyStormMedic` * Linux / Mac OS * `sh CopyStormMedic.sh -tool countRecords -run configFile.copyStormMedic` # CopyStorm Dashboard The Salesforce Job Logging subsystem allows CopyStorm to log job information to the Salesforce CopyStorm Dashboard. This information includes job status, progress, and history. The information that is logged is the same information that can be found in the [CopyStorm Logging Tables](/frequently-asked-questions/how-does-copystorm-work/does-the-copystorm-database-have-logging-tables). The package that must be deployed to Salesforce as part of the [installation and configuration](/installation-instructions/salesforce-job-logging) process includes 3 custom objects, a Permission Set, a Lightning application, a dashboard, and some sample/starter reports. The custom objects and the Permission Set are required for the Salesforce Job Logging subsystem to function. The out-of-the-box fields of the custom objects should be treated as read-only, but additional customization is welcomed. * Custom Objects * CopyStorm Job * CopyStorm Job Table * CopyStorm Exception * Permission Set * CopyStorm Job Logging * Lightning Application * CapStorm * Dashboard * CopyStorm ## Custom Objects The Salesforce Job Logging subsystem will create records in each of the following objects when running a CopyStorm backup job. A reference is included for each object and its fields. ### CopyStorm Job API Name: **CapStorm\_CopyStorm\_Job\_\_c** | **Field Name** | **Data Type** | **Description** | | ---------------------------- | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CopyStorm\_Job\_Id\_\_c | Number(18, 0) | Unique ID for this job assigned by CopyStorm. | | StartTime\_\_c | Date/Time | The time when the job started. Note that this time will be slightly after the start of the CopyStorm executable because it takes a few seconds for CopyStorm to initialize and connect to its data sources. | | EndTime\_\_c | Date/Time | The time when the job completed. | | Transactions\_\_c | Number(18, 0) | The number of Salesforce transactions – API Calls – used during the run (not including logging transactions). | | Transactions\_per\_min\_\_c | Formula (Number) | The number of Salesforce transactions – API Calls – per minute used during the run (not including logging transactions). | | ExceptionCount\_\_c | Roll-Up Summary (COUNT CopyStorm Exception) | The number of exceptions encountered during the run. If this value is non-zero then details for each exception will be recorded in the CopyStorm Job Exception object. | | CopyStormConfigFileName\_\_c | Text(255) | The base name of the CopyStorm configuration file used for the run. | | CopyStormConfigFilePath\_\_c | Long Text Area(32768) | The file path of the CopyStorm configuration file used for the run. | | CopyStormVersion\_\_c | Text(25) | The version of CopyStorm used for the run (e.g. 10.54.3). | | CopyStormDirectory\_\_c | Long Text Area(32768) | The directory in which CopyStorm was installed when the job ran. | | ElapsedTime\_\_c | Formula (Text) | The time taken by the job to run. | | RecordsCopied\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of records copied to the target database. | | RecordsDeleted\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of records deleted from the target database. | | RecordsSkipped\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of records skipped during the run. | | RecordsArchived\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of records archived (Snapshot) during the run. | | FieldHistory\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of field history records created during the run. | | ImagesCopied\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of images copied during the run. | | ImagesRead\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of images read during the run. | ### CopyStorm Job Table API Name: **CapStorm\_CopyStorm\_Job\_Table\_\_c** | **Field Name** | **Data Type** | **Description** | | -------------------- | ----------------------------------------- | -------------------------------------------------------------------------------- | | CopyStormJob\_\_c | Master-Detail(CopyStorm Job) | Reference to the CopyStorm Job record for this table. | | Name | Text(80) | The name of the table/object. | | StartTime\_\_c | Date/Time | The time when CopyStorm began processing the table. | | EndTime\_\_c | Date/Time | The time when CopyStorm finished processing the table. | | ElapsedTime\_\_c | Formula (Text) | The time taken to process the table. | | RecordsCopied\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of records copied to the target database table. | | RecordsDeleted\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of records deleted from the target database table. | | RecordsSkipped\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of records skipped for this table during the run. | | RecordsArchived\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of records archived (Snapshot) for this table during the run. | | FieldHistory\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of field history records created for this table during the run. | | ImagesCopied\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of images copied for this table during the run. | | ImagesRead\_\_c | Roll-Up Summary (SUM CopyStorm Job Table) | The total number of images read for this table during the run. | ### CopyStorm Exception API Name: **CapStorm\_CopyStorm\_Exception\_\_c** | **Field Name** | **Data Type** | **Description** | | ---------------------- | ---------------------------- | --------------------------------------------------------------- | | CopyStormJob\_\_c | Master-Detail(CopyStorm Job) | Reference to the CopyStorm Job record for this exception. | | CopyStormJobTable\_\_c | Master-Detail(CopyStorm Job) | Reference to the CopyStorm Job table record for this exception. | | ExceptionText\_\_c | Long Text Area(32768) | The message included with the exception. | | Traceback\_\_c | Long Text Area(32768) | The traceback included with the exception. | ## Permission Set The included permission set is required for the logging subsystem to function as it gives the assigned user Read, Create, Edit, Delete, View All, and Modify All permissions for the custom logging objects specified above as well as access to the CapStorm Lightning Application and CopyStorm Dashboard (along with the included starter/sample reports). * Permission Set Name: **CopyStorm Job Logging** * API Name: **CapStorm\_CopyStorm\_Job\_Logging** # CopyStorm/SanityChecker **As of CopyStorm/Medic 10.54.2, the Base64 Sanity Checker function has been merged into CopyStorm/Medic.** **For the latest documentation, please see:** * [base64-sanity-checker](/reference/copystorm-medic/base64-sanity-checker) *** When the Salesforce REST API returns a record with a BASE64 column, the BASE64 data is returned as a hyperlink that can be used to read the raw bytes. Experience has shown that it is possible for the hyperlink returned by Salesforce to return a HTTP 404 (not found) or an HTTP 500 (internal lightning platform error) rather than the expected HTTP 200 (OK). When this happens: * The BASE64 data cannot be read by the REST API. * The same operation using the SOAP API results in an UNKNOWN\_EXCEPTION or an unparsable SOAP body The CopyStorm/SanityChecker tool tests the hyperlinks returned by the REST API and reports the ID of records where the hyperlink fails (e.g. Does not return a HTTP 200). It is often wise to check each reported record using the Salesforce GUI to make sure that the data is really available. If you find a corrupted Attachment record report it to Salesforce. ## Step 1: Enter Salesforce Credentials ![](/files/2DnlgamMEXZlpmXyHGkj) ## Step 2: Set Search Parameters The only required parameters are the tables to search, but there are a number of other parameters you may want to modify. | Parameter | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Modified After | Search for records starting at this date. A date can be selected from the menu or typed in directly. Values are restricted to datetime formats supported by SOQL. | | Modified Before | Search for records up to this date. A date can be selected from the menu or typed in directly. Values are restricted to datetime formats supported by SOQL. | | Salesforce Batch | Number of records to process from Salesforce in a single batch. The default usually provides optimal performance. | | # Checker Threads | Number of base 64 hyperlinks to analyze concurrently. | | Max Timeout (sec) | Number seconds to allow for each hyperlink test. If a hyperlink HTTP request times out it is assumed to be OK based on Capstorm’s experience with problem records. Setting this to a smaller value may improve the overall throughput of the analysis. | | Report Timeouts | If this option is checked then all HTTP timeouts will be reported as potential problems. Enabling this feature may be useful in selecting an optimal *Max Timeout* value for your organization. | ![](/files/cb38lfgSpSTVGQ3lBvUj) ## Step 3: Check Hyperlinks The next step is to click on the *Start Sanity Check* button and wait for the analysis to complete. Note: * Each hyperlink check takes one Salesforce API call. There is one additional API call for every 2000 records. * An running analysis can be cancelled but the cancellation process may not be immediate. The reason is that pending HTTP requests must complete or timeout before the cancellation process is finished. While the sanity check is in process, the tool will look like the following: ![](/files/EyOaw2togD9oGoOtEJDw) The following example shows what a report of potential problems looks like. You can copy/paste the potential problems into your favorite editor. ![](/files/laM66mYOBEp5T1PZ4Qm9) ## Summary Hopefully when you scan your Salesforce the tool will not find potential problems. However, if potential problems are found then it is wise to check the validity of each record using the Salesforce GUI. If you are using CopyStorm to perform backups, then this class of error will be discovered during the normal course of a backup. Unfortunately the error is often masked by an UNKNOWN\_EXCEPTION or an Invalid SOAP Envelope response from Salesforce. --- [Next Page](/llms-full.txt/1)