Miscellaneous and Rarely Used Options
The Miscellaneous and Rarely Used parameter sections control various aspects of CopyStorm.
Miscellaneous Options
| Parameter Name | Default Setting | Required | Description |
|---|---|---|---|
| Show All Options | No | No | If checked, all CopyStorm options will be displayed. If unchecked, only commonly modified options will be displayed. |
| Log Format | Full | No | This option controls the amount of detail sent to the CopyStorm log file. Options include:
|
| Use Apache Logging | No | No | If checked, logging will be performed with the Apache Commons Logging framework. By default, Log4j will be used. |
| Config Directory | No | By default, CopyStorm will look for configuration file overrides in the directory “config” within the CopyStorm installation directory. If this value is set then CopyStorm will also look in this directory for configuration file overrides. | |
| Estimate Total Time | No | No | If checked and CopyStorm is running with a GUI, an extra SELECT query will be issued for each table to determine the total number of records to be copied. |
| Debug Mode | No | No | If checked, CopyStorm will output logging messages to the debug console (normally hidden). |
Rarely Used Options
The following options change the normal CopyStorm algorithm for fetching records from Salesforce. Do not use these options for normal incremental updates. CopyStorm normally fetches only records that have been added or modified based on the timestamp data stored in the local database.
These options force CopyStorm to fetch all records based on these parameters. When used in conjunction with the “Force Updates” option a corrupted database can be repaired without truncating and reloading.
| Parameter Name | Default Setting | Required | Description |
|---|---|---|---|
| Modified Since | No | If a date is specified, then only records created or modified on or after the specified date are copied. Built in relative dates are available from a pull-down menu or a fixed date can be entered. When entering a date, the interface will have a yellow background which clears when a date is recognized.
Fixed date formats are:
|
|
| Modified Thru | No | If a date is specified, only records created on or modified on or up to the specified date are copied. See format notes above. | |
| Ignore Timestamps | No | No | If checked, CopyStorm will retrieve records in ID order rather than timestamp order.
WARNING: unless a CopyStorm job runs to completion without errors, using this option will make subsequent incremental updates fail to fetch the proper records. |
| Rebuild on DML Errors | No | No | If checked, CopyStorm will drop and rebuild a table if an ALTER statement on the table causes a DML exception. |
| Database Logs in UTC | No | No | If checked, timestamps recorded in CopyStorm’s job tracking log tables will be in UTC rather than local time. |
| Internal Parameters | No | Launches a dialog used to modify internal CopyStorm parameters that should not be modified without instructions from Capstorm. |
Internal Parameters
| Parameter Name | Default Setting | Required | Description |
| Update Using DELETE/INSERT | No | No | If checked then CopyStorm will UPDATE records by first deleting them and the re-inserting them. This option is ignored if the Optimize Update SQL option is checked. |
| AnyType Length | 255 | No | Allocate 255 characters for Salesforce columns using the AnyType data type. Use of an AnyType column is rare and only seen in Salesforce system level tables. |
| DDL Timeout | 3600 | No | Set the number of seconds CopyStorm will wait for a DDL operation to complete. DDL operations, like CREATE INDEX, may take considerably longer than DML operations and thus have their own timeout parameter. |
| Optimize Timestamps | Yes | No | If checked, then use the time of the most recently complete CopyStorm job to reduce the number of records read from Salesforce.
If not checked, then the timestamp of the newest record in Salesforce is used to build SOQL queries, and at least one record from each table will always be read. |
| Ignore Numeric Overflow | Yes | No | If checked, then if a numeric formula column contains a value which would cause a numeric overflow in the database then the database value will be set to null. The most common cause of this issue is an ill-defined Salesforce formula column with a near divide by zero error. |
| Backup Images | TRUE | No | If checked then CopyStorm will replicate images embedded in Salesforce Rich Text fields and Knowledge Articles. Images are stored in the tables CopyForceRtImage and CopyForceRtImageData. |
| Optimize Deleted Record Scan | TRUE | No | If checked then CopyStorm will use information from the previous backup to minimize the number of records scanned when searching for deleted records. This option is ignored if the Delete Older Than option is blank. |
| Rows per Multi Insert | NULL | No | If specified then multiple rows will be sent to the database in a single INSERT statement. Please note that not all databases support this feature and when the database is on a local network there is unlikely to be any performance gain by enabling this feature. |
| Create Indexes Early | TRUE | No | If checked then CopyStorm will create indexes on Salesforce reference fields when a table is initially created. If not checked then reference field indexes will be created after the first backup of a table. This option can avoid a long delay or timeout when indexes are originally created on tables with many rows. |
| Create IsDeleted Index | TRUE | No | If checked then CopyStorm will create an index on the fields (IsDeleted, SystemModstamp). |
| Obfuscate Schema | If checked then CopyStorm will obfuscate database table and column names. | ||
| Silent Mode | No | No | If checked, CopyStorm will only display errors to stderr when run in batch mode. |
| SFDC Commit Latency | 120 | No | The number of seconds a Salesforce records has to be unchanged until it can be trusted that records have been written to Salesforce in timestamp order. This is a workaround for Salesforce’s belief in time travel — a record with a timestamp of 12 noon can be written to Salesforce before one with a timestamp of 11:59 m. |
| Timestamp Latency | 2 | No | When searching for records in Salesforce based on the timestamp of the most recent record in CopyStorm then subtract this number of seconds from the most recent timestamp in Salesforce. |
| Row LIMIT per Query | 20000 | No | The default number of records to read from Salesforce with a single SOQL statement. A larger number may reduce the number of Salesforce API calls slightly but also may result in query time outs that CopyStorm must recover from using additional API calls. |
| Always Validate SOQL | No | If checked then CopyStorm will validate every generated SOQL statement to insure that a QUERY_TOO_COMPLEX error is not generated. Since CopyStorm dynamically recognizes QUERY_TOO_COMPLEX errors and rewrites queries to avoid them there is little need for this parameter to be set unless the user is a CAPSTORM developer. | |
| Find Optimal SOQL | No | If check then CopyStorm will take extra effort to generate to optimal load balanced SOQL statements when multiple SOQL statement are needed to read a single row from a table. This parameter is for CAPSTORM developers. | |
| Max Simple Query | No | The smallest SOQL query that CopyStorm will assume never will generate a QUERY_TOO_COMPLEX error. This parameter is for CAPSTORM developers. | |
| Analyze UNKNOWN_EXCEPTION | No | No | When Salesforce returns an UNKNOWN_EXCEPTION this generally indicates that a SOQL query failed in a way the Salesforce infrastructure did not expect. When this option is checked and CopyStorm is running in GUI mode, CopyStorm will do a rather expensive analysis of the failed SOQL and attempt to determine a very small SOQL statement which exhibits the problem. The very small SOQL statement is what Salesforce support will require (even demand!). |
| Enable SOQL Query Lookahead | Yes | No | When checked, CopyStorm will read the next batch of record for a table while processing the current batch. This improves the overall performance of CopyStorm with a slightly larger load on the local computer. |
| Validate SOQL End of Data | TRUE | No | If checked then CopyStorm will verify that the backup of a table is complete by running a single validation query. This is necessary in cases where SOQL returns no records from a query when there are records available. |
| Max Base64 Reader Threads | 5 | No | Set the maximum number of base64 values that will be read concurrently when processing tables containing base64 columns using the REST API. This option can greatly increase the rate at which tables like Attachment and ContentVersion are backed up by downloading the raw file content for multiple records concurrently. |