Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Upgrading CopyStorm can be a simple 30 second task if installed using a specific directory configuration.
The directory structure we recommend for a CopyStorm installation looks like the following:
CopyStorm
Created by unzipping the CopyStorm download (see the Installation Instructions).
CapstormCustomerKey.txt
Capstorm Customer Key file used for licensing.
ProductionA
ProductionA.copyStorm
CopyStorm configuration file for backing up the ProductionA Salesforce.
ProductionALessFrequent.copyStorm
CopyStorm configuration file for backing up the ProductionA Salesforce. This configuration file may backup Salesforce tables that are more expensive to backup (examples: Attachment, ContentVersion).
ProductionA.copyStormMedic
CopyStorm/Medic configuration file for maintaining the ProductionA Salesforce backup.
config
XML configuration files used by CopyStorm when backing up the ProductionA Salesforce.
ProductionB
ProductionB.copyStorm
CopyStorm configuration file for backing up the ProductionB Salesforce.
config
In other words:
Create a directory next to the CopyStorm installation called CopyStormConfig to hold all of your site specific configurations and customizations.
Create a separate subdirectory under CopyStormConfig for each SalesForce instance to be backed up.
If you are managing your own job scheduling rather than using CopyStorm/Director then the BAT or sh files used to run your scheduled jobs should be stored with their respective CopyStorm configuration files. If you are using CopyStorm/Director the CopyStormConfig directory works perfectly for configuring CopyStorm/Director.
If CopyStorm is installed using the directory structure detailed above, this process will upgrade the installation to a new version:
Remove or rename the CopyStorm directory.
Unzip the new CopyStorm release in the same location as the old CopyStorm directory.
This process should end with a replacement CopyStorm directory containing an upgraded version of the CopyStorm software.
Many sites will decide to backup their Salesforce using multiple CopyStorm jobs. Why?
A site will want the data in specific tables in the CopyStorm database to be within minutes of the data in Salesforce. These tables in CopyStorm are frequently used for analytics or operational reporting.
Some tables in Salesforce simply take longer to backup and are not important to keep within minutes of Salesforce. Common culprits include Attachment, ContentVersion, and Knowledge Article version tables (each of these tables contain base64 columns and take a single API call to read a record).
If you are not content with a single CopyStorm job then this article describes a few best practices for using multiple CopyStorm jobs.
A common setup strategy involves dividing the tables in Salesforce into three separate buckets.
XML configuration files used by CopyStorm when backing up the ProductionB Salesforce.
Tables to backup frequently.
Tables that are expensive to backup and can be backed up less frequently.
Metadata.
For this setup, we create three separate CopyStorm configuration files and schedule them to run a different times and frequencies.
This configuration file is will be run frequently and backs up tables we want to keep in close agreement with Salesforce. Rather than specify what tables are in the frequent category, we specify the tables to exclude. Many sites will run this backup every 15 to 30 minutes.
This configuration file will be run infrequently and backs up tables that do not need to be kept within a few minutes of Salesforce. Use the “Include List” CopyStorm feature to specify the tables to back up. Many sites run this backup every few hours.
This configuration file will be run infrequently and backs up just Salesforce metadata. Use the “Include List” CopyStorm feature to tell CopyStorm to backup up just a single table — in this case the table specified is not in Salesforce.
Once multiple CopyStorm configuration files have been created, jobs need to be created and scheduled.
Store the CopyStorm configuration files using .
Create a script for running each job using the .
.
There is no functional limit to the number of CopyStorm jobs you can use to backup Salesforce and some sites will use four or more jobs — usually to support keeping key CopyStorm tables within minutes of Salesforce. The choice is really yours.
Here are few suggestions when running many jobs.
Try to avoid backing up the same table in multiple jobs. Though this will not damage the integrity of the backup it will often generate annoying messages.
Consider using a “super scheduler” tool like to manage scheduling and logs. In general a super-scheduler make managing a lot of backup jobs and their logs much simpler.
Be aware that a single Salesforce Login Name can have at most 20 concurrent active queries. If you have more than 20, Salesforce may silently kill one of your sessions. Though CopyStorm will recover from this unfriendly Salesforce action automatically, many applications will not behave so nicely.
The simplest way to setup a new database for CopyStorm in MySQL is:
CREATE SCHEMA copystorm
However, it is important that your database server be configured to support the full set of UTF-8 characters supported by Salesforce. Default MySQL configuration typically uses the latin1 character set, which omits UTF-8 characters like a smiley face with sunglasses (U+1F60E). If a non-included character appears in Salesforce, MySQL will refuse to accept it and report an error like:
SQL state [HY000]; error code [1366]; Incorrect string value: ‘\xF0\x93\x83\xA2\xF0\x93…’
You can tell if MySQL is already configured correctly by running the following command:
SHOW VARIABLES LIKE ‘character_set_server’
If the current value of “character_set_server” is “utf8mb4” then your server is setup correctly. If not, here are the steps to correct the problem:
Open the mysqld.cnf file for your MySQL. This contains the default parameters for your server.
Look for the “character_set_server” setting. If you find it then change it to utf8mb4 and save the changes. If not, then add lines like the following to the “[mysqld]” section:
character_set_server = utf8mb4
character-set-server = utf8mb4
MySQL Workbench often uses a default character set of utf8. If a string containing 4-byte utf8 characters is viewed in Workbench then the 4 byte characters appear as question marks. To fix this behavior, set the character set for the session:
SET CHARACTER SET utf8mb4;
If your workbench session still does not correctly display the characters then it may be a workbench problem. We have seen cases where Heidi (the MariaDB Workbench equivalent) and SQuirrel display the data correctly and Workbench does not.
Instance page size is set (using the variable innodb_page_size) when you first create a MySQL instance. This can cause problems when Salesforce contains very wide tables whose records will not fit into a single page (by default 16KB), and cannot be changed without deleting and rebuilding the database.
Capstorm always recommends using a 64KB page size when setting up a Capstorm MySQL database.
MySQL’s maximum log file size needs to be around 4 times the size of the large object in MySQL. If this is not the case you may see a “Row Size Too Wide” error from MySQL. To change your log file size search for the “innodb_log_file_size” in the MySQL documentation.
The default InnoDB format, Antelope, stores the first 768 bytes of a TEXT column with the row data — the rest is stored elsewhere. Since Antelope is the default InnoDB format almost every MySQL instance will be using it.
If at runtime a row has a lot of TEXT columns and many of them have more than 768 bytes, you may see a “Row Size Too Wide” error message from MySQL. An easy way to avoid this issue is to use the InnoDB Barracuda format and change tables to use ROW_FORMAT=Dynamic.
To change a database to use the Barracuda format:
SET GLOBAL innodb_file_format = barracuda;
SET GLOBAL innodb_file_format_max = barracuda;
This will cause future tables to use the Barracuda format. To force a table to use a Dynamic row format:
ALTER TABLE tableName ROW_FORMAT=Dynamic;
In addition, you can force all tables created by CopyStorm to be Dynamic Barracuda tables by adding the following lines to the config file “TableRuleRegistry.xml”:
While achieving CopyStorm performance can vary from company to company, the principles and practices in this article should help you optimize CopyStorm for your organization.
In most cases a company will want to optimize the rate at which records are backed up from Salesforce. In this case there are typically only a few parameters that typically yield performance improvements which:
Increase the rate at which data arrives from Salesforce.
Increase the rate at which your database can accept changes.
The best way to improve the rate at which data arrives from Salesforce is to:
Increase the number of tables CopyStorm backs up at the same time. This value is controlled by a parameter called “# Salesforce Threads” under “Salesforce API” on the “Configuration” tab. Most companies will see optimal performance using a value of 3 to 10. The optimal value for your company will likely depend on the rate at which your database will accept data.
People often ask “Why not use the BULK API since it is the fastest approach?” The short answer is:
Though the execution time reported by the BULK API is short, the amount of time a human has to wait (wall time) for a copy job to complete is longer than using multiple SOAP or REST threads. Ten SOAP threads will beat ten BULK jobs almost 100% of the time. We suspect that this is because BULK jobs run at a much lower priority in Salesforce than SOAP or REST requests.
There are two common cases where memory usage is a problem:
Because the Salesforce SOAP API returns Attachments inline the memory requirements for large SOAP attachments will be almost four times the size of the attachment reported by Salesforce. There are two primary ways to solve this problem:
.
If using Oracle or SQL/Server switch to the REST API. In this case large Attachments will no longer be kept in memory and will be streamed directly to the database. The REST API helps slightly for other databases but not dramatically, as they require large objects to be kept in memory
If you are forced to work on equipment that does not have much memory (e.g. an old 32-bit 2GB Windows box) then:
Huge attachments may not be possible (unless using Oracle or SQL/Server).
CopyStorm memory use can be limited at the cost of more Salesforce API calls by reducing the “Salesforce Batch Size” parameter.
CopyStorm typically has very modest compute requirements — if you find yourself in a situation requiring more memory or compute than available Capstorm generally recommends upgrading hardware (a standard desktop has more than enough resources to run CopyStorm).
If your CopyStorm database is heavily used for business analytics, operational reporting, or other purposes while a CopyStorm backup is running you may experience database performance issues due to lock contentions. The easiest way to solve this problem is to reduce the number of records sent to the database in a single transaction:
CopyStorm provides a robust data retention facility that allows Salesforce records to be recoverable at virtually any point in time, and provides a lot of flexibility in establishing a data retention policy. What is a data retention policy? A data retention policy is a set of practices that retain historical corporate data for a specified period of time and, in the case of databases, allows historical data to be restored. A good data retention policy supports tasks like “Restore this data to how it was last Wednesday”.
There are two ways to configure a data retention policy within CopyStorm:
Using CopyStorm’s built-in “Default Point-In-Time” Snapshot Policy to capture virtually any record change that may be needed for restoration later.
Creating a custom data retention Snapshot data source, capturing a targeted subset of data for later restoration.
The rest of this document provides information useful for setting up CopyStorm’s built-in “Default Point-In-Time” Snapshot Policy. For more comprehensive details on how to configure a custom Snapshot Policy to suit specific data retention needs, please see the documentation for .
CopyStorm’s built-in “Default Point-In-Time” Snapshot Policy is preconfigured to record all data changes that may be required for recovery.
To enable CopyStorm’s “Default Point-In-Time” Snapshot Policy:
Go to CopyStorm’s Snapshot tab.
Click on the “Edit” link for the Default Point-In-Time Snapshot Policy.
Select the “Active” checkbox.
This will enable the snapshot, after being enabled CopyStorm will automatically capture record changes in the Default Point-In-Time data source.
Which line depends on the exact MySQL version and OS distribution you are using.
Restart MySQL.
Run SHOW VARIABLES to verify the change.
<TableRule name="default" >
<CreateTableRule database=”MySQL” option=”CHARACTER SET utf8mb4 ROW\_FORMAT=Dynamic” />Set the # Days to Retain record changes.
All record changes will be stored in the database for the specified number of days.
Save changes.
To get the most out of your PostgreSQL database there are a number of tasks you will want to do after you have installed the PostgreSQL Server
The default maximum memory allocated to a default PostgresQL is minuscule. Yes, this means that a stock PostgreSQL will run on a computer with less capability than a modern cell phone but it also means a lot of unnecessary disk swapping. There are a handful of parameters you will want to bump — all found in postgresql.conf.
The following table lists a few key default PostgreSQL parameters and the parameters used in Capstorm’s labs.
For additional help see:
This article focuses on specific technical decisions that need to be made when setting up an Oracle database for CopyStorm. If you need more general Oracle help then grab a DBA — Oracle shops always have them.
In short, use AL32UTF8. Salesforce uses the full extended UTF-8 character set (up to 4 bytes per character) and an equivalent character set should be used in Oracle. Do not use UTF8 even though your backup will almost always work since, despite its name, the Oracle UTF8 set is not a full implementation of the UTF8 standard.
By default CopyStorm will use VARCHAR2 columns until the declaration of a column exceeds 4000 bytes. After 4000 bytes CopyStorm will use a CLOB column type. If you have a version of Oracle that supports the max_string_size=EXTENDE D property, extending VARCHAR2 lengths to 32767 bytes, you should consider forcing CopyStorm to use this feature. Unfortunately, since automatically detecting EXTENDED support cannot be done by a regular Oracle user but requires nearly the same permissions as SYSTEM, CopyStorm must be configured to use the feature by following the directions in this article:
CopyStorm automatically detects the maximum length for names at run-time and will use 30 or 128 long names depending on what your Oracle version supports. In all cases the mappings between Salesforce table and field names are kept in tables in the CopyStorm created database. See this article for the details.
Capstorm licensing is implemented using a unique Customer Key. The purpose of a Customer Key is to uniquely identify a customer to the Capstorm licensing system, so the licensing system can implement policies unique to a customer.
The Customer Key is associated with a single organization’s use of Capstorm software and is used to determine which applications and features may be used.
The only way to acquire a customer key is to contact your Account Executive or . They will provide you with a customer key contained in a file named:
CapstormCustomerKey.txt
You have a number of choices when determining where to install the customer key file:
If you have followed and installed your Capstorm products in a directory structure like:
There are a number of quirks to MySQL when it comes to adding columns or altering columns in an existing table. The technique it employs, unlike other databases supported by Capstorm, can be quite time-consuming.
To perform an ALTER TABLE abc ADD COLUMN command, MySQL:
Creates an empty shadow of the original table WITH the new column.
Copies the entire existing table to the shadow table.
The approach taken when setting up a CopyStorm/SQL Server database may go a long way in reducing long-term ownership costs. There are two primary recommendations that should be followed when configuring a CopyStorm/SQL Server database.
The most common “mistake” made when setting up a SQL/Server database for CopyStorm is to use a Domain User along with a user specific schema for storing Salesforce data. For example,
The database administrator creates a database named “salesforce”.
User CAPSTORM\greg runs CopyStorm using integrated security (e.g. a domain user login) and creates all database tables in the [CAPSTORM\greg] schema.
shared_buffers
128MB
4GB
work_mem
4MB
1GB
maintenance_work_mem
64MB
1GB
effective_cache_size
4GB
12GB
/opt/Capstorm/CopyStorm/
/opt/Capstorm/CopyStormRestore/
/opt/Capstorm/CopyStormMedic/
/opt/Capstorm/CapstormConfig/
The key file can be stored at:
/opt/Capstorm/CapstormConfig/CapstormCustomerKey.txt
Once installed, all Capstorm applications will find it and use it.
If the customer key file is stored in the root directory of any Capstorm application, the application will find it and use it when talking to the Capstorm license server.
For example, if CopyStorm is installed at “/opt/Capstorm/CopyStorm”, the key file can be stored at:
/opt/Capstorm/CopyStorm/CapstormCustomerKey.txt
The key file may also be installed in the user’s home directory:
Windows: C:\Users*Username*\CapstormCustomerKey.txt
Linux: /home/Username/CapstormCustomerKey.txt
Replaces the existing table with the shadow table.
This MySQL behavior can make it faster to rebuild a CopyStorm database when upgrading protocols rather than upgrading a MySQL database in place.
CopyStorm version numbers contain three parts:
Major Version
Protocol Version
Minor Version
For example, 7.34.1 is major version 7, Salesforce protocol 34, and minor version 1. A value of 7.31.1 would indicate the same feature set but using Salesforce protocol 31.
When Salesforce updates protocols new columns and tables are almost always added (this happens 3 times a year, CopyStorm releases a new protocol update after each Summer protocol update). The first time you run CopyStorm with an updated protocol on your MySQL database new tables will be added (not a problem) and new columns may be added to existing tables (a problem for large tables).
There is no hard and fast “best practice” but there are a few options that have various tradeoffs.
If you are keeping deleted records forever or preserving deleted columns then your only option is to let MySQL upgrade the tables in place. Other approaches would cause you to lose data.
If you do not keep deleted records and your organization can live with a table being offline for a small amount of time, the easiest option is to run CopyStorm once with the “Force Rebuild” option checked so tables can be fully rebuilt. This option will make CopyStorm drop and rebuild each table when backing up Salesforce, and can be found on the “Advanced” tab. This option should always be turned off when performing incremental backups — dropping and rebuilding each table each time a backup runs defeats the purpose of being incremental.
If you do not keep deleted records but your organization wants the MySQL database down for the minimum amount of time, then:
Use the new version of CopyStorm to create a new database while continuing to run the older version of CopyStorm on the older database.
When the new version of CopyStorm is done backing up Salesforce to the new database, use it to replace the older database. Use the new version of CopyStorm going forward.
His default schema is [CAPSTORM\noah].
All SQL table references have to include [CAPSTORM\greg].
Though a database administrator can get around this problem by making the default schema be [CAPSTORM\greg] for all CopyStorm database users, this is not a great idea. The core problem with this process is that the ownership of shared Salesforce backup data is tied to a real person — when that person wins the lottery or moves to a new position someone else will need to own the data.
Core Principle: The owner of a SQL/Server Salesforce database should not be a Windows Domain Account.
Capstorm considers the following to be the best practice for creating and populating a CopyStorm database in SQL/Server:
Create a new SQL/Server database.
The only purpose of this database is to hold CopyStorm created backups of Salesforce.
Create a non-domain SQL/Server user. Commonly used names include “salesforce”, “copystorm”, or “capstorm”.
Give this user ownership of the dbo schema in the CopyStorm database.
Make the default schema “dbo” in the CopyStorm database.
Always run CopyStorm using the non-domain user.
Create a group with read-only access to the salesforce database with a default schema of “dbo”.
When a domain user needs access to the SQL/Server Salesforce database, make them of a member of the group created in the previous step.
If your organization is smaller, the only essential best practice steps are 1-3.
If your organization chooses to use domain accounts for all SQL/Server authentication that’s OK, with the caveat that the CopyStorm database will require permission updates when the person tied to the domain account decides to retire.
Performance of CapStorm applications can be greatly impacted by the SQL Server query planner. By default, the “parameterization” parameter is set to “SIMPLE” – which can cause common CapStorm queries to not be fully optimized by the SQL Server query planner.
It is recommended that you set the “parameterization” parameter to “FORCED” – this will enable SQL Server to better optimize query plans for common CapStorm application queries.
You can find out if parameterization is forced by running this query:
SELECT name, is_parameterization_forced FROM sys.databases WHERE name=DB_NAME()
And to enable forced parameterization, you can run this query:
ALTER DATABASE CURRENT SET PARAMETERIZATION FORCED
The purpose of this page is to document how to configure CopyStorm desktop apps to require login via OAuth 2.0 identity provider.
This feature updates CapStorm applications to require a login prior to accessing the application GUI – this feature is separate from features enabling OAuth for Salesforce or database login.
OAuth login for CapStorm desktop applications is only supported on Windows.
Download the plugin.
OAuth application authentication requires a plugin to be installed into CapStorm desktop applications.
Contact your CapStorm account representative to obtain the plugin zip file.
Install the plugin.
Extract the OAuth2 plugin zip file to the plugin folder in the CapStorm application installation directory.
The plugin is a sub-folder of the main application installation directory – i.e. it is next to the “config” folder and “CopyStorm.bat” file.
OAuth settings are stored in a file named AuthParams.xml in the CopyStorm config/ directory.
The specification for this file is:
Request: Parameters to configure OAuth requests.
url: the IdP’s authorization endpoint.
Param elements under the Request element are added to the OAuth HTTP request.
Validate: Parameters to configure OAuth validation requests.
Make sure the redirect_uri matches the one registered in the OAuth IdP.
The CapStorm OAuth plugin requires token validation – if the OAuth request or token validation fails, then the application will not launch.
When this feature is configured, CopyStorm will perform the following steps before displaying the application UI:
Check for a valid AuthParams.xml file.
Launch a browser, navigating to the IdP login page.
Wait for the authorization code callback.
Validate the authorization code using the token endpoint.
If the OAuth login fails, the token request fails, the token fails validation, or the AuthParams.xml file is misconfigured – CopyStorm will not launch. Instead of launching the application, CopyStorm will either display an error window or write the error message to STDOUT / STDERR. If you do not see an error message, try launching CopyStorm from a CMD window as “CopyStorm.bat -debug”.
If assistance enabling this feature is needed please reach out to your CapStorm account representative or .
By default, CopyStorm inserts/updates records in the Salesforce backup database when the corresponding record timestamps change in Salesforce. When Salesforce formula fields are involved, a field value may change without the timestamp on a record changing. This article describes several mechanisms provided by CopyStorm to keep database columns in sync with the corresponding formula columns in Salesforce.
Salesforce supports two classifications of formula fields:
Row-based formulas
Determine their value based on the values of other columns/fields in their row/record.
Values for this type of formula change only when another field in the same record changes.
Variant formulas
Determine their value based on data found in other records (data stored anywhere but the same row/record).
This classification of fields includes:
Summary fields.
Maintenance of Row-Based Formula fields is fairly simple because there are only three conditions where the value of the formula field can change:
A new formula field is created.
In this case all records modified AFTER the formula field is created will automatically be updated in CopyStorm.
Records whose last modification date is earlier than the formula creation date must be manually refreshed.
The definition of an existing formula field is modified.
The following scenario illustrates the problem to solve:
Suppose:
The Account table has 100 existing rows and no formulas.
CopyStorm has just copied all 100 rows to a database.
A programmer adds a new formula column named “age__c” using the formula “1+2”.
Our recommended best practice is to force an update on a Row-Based Formula column any time:
A Row-Based Formula column is created.
A Row-Based Formula column is modified.
Forcing a column update only takes a few minutes (in the GUI or as a batch job) using the tool. Alternatively, the tool can be used to refresh every formula in a table.
Another solution is to truncate or drop the corresponding database table and let CopyStorm rebuild it during the next backup.
The best practice for Row-Based Formula columns is to run on the column when it is created and any time the formula definition changes.
A Variant Formula field has a value that changes even when the data in its corresponding row has not changed. These changes may occur every minute, hour, month, or irregularly. Examples of Variant Formula fields include:
The number of minutes a support case has been in an open state.
The number of hours it took for an account rep to open a new lead.
The number of months since a customer was last contacted.
The total amount of bookings from an account in the current year.
In every case, keeping a CopyStorm database in sync with Variant Formula fields often requires a business process decision to be made for each field.
The best practice is to:
Divide Variant Formula fields into different buckets based on how frequently their value in CopyStorm should be synced with Salesforce.
Create a CopyStorm/Medic configuration file for each bucket of formula fields. Each configuration file should be configured to update specific columns using the tool.
Create a scheduled job for each CopyStorm/Medic configuration file.
If you have a lot of CopyStorm/Medic jobs to run due to having a complex set of Variant Formula fields then you may want to consider . CopyStorm/Director is designed to manage and track the scheduling and status of many CopyStorm and CopyStorm/Medic jobs.
In CopyStorm there is a lot of flexibility in how field history tracking is setup. Specifically:
Field history tracking is available for all fields in all tables.
The number of days fields history is kept is controllable on a per table basis.
Field history can be kept forever.
Though it is not difficult to set up field history rules, there are a handful of questions that need to be answered.
Do I want to control what tables are backed up using a Whitelist or Blacklist approach?
By default, how many days should field history records age before they are automatically deleted by CopyStorm?
Are there any tables whose field history records I want to keep longer (or shorter) than the default?
The rest of this document provides examples for both a Blacklist and Whitelist approach.
In a blacklist approach to field history the tables that SHOULD NOT be tracked must be specified. In addition to deciding on what tables to be excluded there are two other decisions to make:
By default, how many days should field history record be kept before being deleted from CopyStorm?
Are there any tables whose field history records should be kept longer (or shorter) than the default?
For the following example:
By default, keep field history records for 90 days.
Exclude Salesforce History, Feed, and EventLog records from field history tracking.
Keep Account, Contact, Case, and Email fields history changes for 800 days.
In a whitelist approach to field history tracking, all tables to be tracked are explicitly specified. The advantage is precision while the disadvantage is that any new tables added to Salesforce will not participate in field history tracking by default.
In addition to deciding on what tables to include there are two other decisions to make:
By default, how many days should field history record be kept before being deleted from CopyStorm?
Are there any tables whose field history records should be kept longer (or shorter) than the default?
For the following example:
By default, keep field history records for 90 days.
Include a lot of tables.
Keep Account, Contact, Case, and Email fields history changes for 800 days.
Though CopyStorm will faithfully back up records that have been deleted in Salesforce, there are a few cases where Salesforce will delete records without the deletion being visible to CopyStorm. These situations occur when:
A hard delete is performed in Salesforce.
A Salesforce table does not track deleted records (this occurs in system tables).
When a Salesforce record is deleted, Salesforce marks the record as deleted by setting the “isDeleted” field to true and by putting the record in the recycle bin. When CopyStorm runs it retrieves all updated records — including those which have been marked as deleted. This is used to record record deletion in the CopyStorm database, but there are a few cases where a record is deleted and not put in the recycle bin.
This is called a hard delete. When this happens, the normal CopyStorm update flow will never determine that the record has been deleted.
Fields computed from a changing value (e.g. “CloseDate – TODAY()”)
In this case all records modified AFTER the formula field is created will automatically be updated in CopyStorm.
Records whose last modification date is earlier than the formula creation date must be manually refreshed.
The data used to compute the formula field changes.
Since the data used to compute a Row-Based Formula field is in the same row as the formula, this will change the modification timestamp on the record and changes will automatically be reflected in CopyStorm.
CopyStorm runs and it:
Creates a new database column named “age__c”.
Makes no changes to the database (because no records have been updated there are no modification dates later than the previous CopyStorm backup).
The age__c column defaults to NULL for all records in the CopyStorm database.
A user makes a change to the account ACME.
CopyStorm runs and it:
Detects that the ACME record has changed and updates the database.
Sets the age__c value for ACME to “3”.
No other Accounts are updated and retain NULL for age__c (because none have modification timestamps later than the previous CopyStorm backup).
A programmer modifies the age__c column and changes the formula to “1+2+3”.
A user makes a change to the account BIOSPHERE.
CopyStorm runs and it:
Detects that the BIOSPHERE record has changed and updates the database.
The BIOSPHERE record has a value of “6” for the age__c column.
The ACME record retains the value of “3” for the age__c column.
All other Accounts retain the value of NULL for the age__c column.
url: the IdP’s token validation endpoint.
Param elements under the Validate element are added to the token validation HTTP request.
Be sure to update the following parameters inside the AuthParams.xml:
client_id
client_secret
redirect_uri
Continue startup only if authentication succeeds.
<AuthParams>
<Request url="https://your-oauth2-host/oauth2/authorize">
<Param name="client_id" value="YOUR_CLIENT_ID"/>
<Param name="redirect_uri" value="http://localhost:12345/callback"/>
<Param name="response_type" value="code"/>
<Param name="scope" value="openid email profile"/>
</Request>
<Validate url="https://your-oauth2-host/oauth2/token">
<Param name="grant_type" value="authorization_code"/>
<Param name="client_id" value="YOUR_CLIENT_ID"/>
<Param name="client_secret" value="YOUR_CLIENT_SECRET"/>
<Param name="redirect_uri" value="http://localhost:12345/callback"/>
</Validate>
</AuthParams>There are four primary conditions where hard deletes occur:
A user manually empties a recycle bin.
A record has been in the recycle bin for a long time (weeks) and Salesforce automatically deletes it from the recycle bin.
Two records are merged.
Salesforce deletes a record without using the recycle bin. This does not occur on most tables, but this article identifies the handful of tables with this issue.
To solve this problem, the “isDeleted” field needs to be updated to reflect records that have been hard deleted in Salesforce. The CopyStorm/Medic fixIsDeleted tool has been designed to do this exact task.
How often (and if) your site resolves hard deletes is up to you. If you would like the CopyStorm backup to reflect hard deletes then the following list may help — it contains known tables that can contain hard deletes in a out-of-the-box Salesforce instance:
Table Name
Common Impact
Description
OpportunityLineItem
High
The operation of syncing an opportunity with a quote does a hard delete on existing line items.
OpportunityContactRole
Because hard deletes can be caused by a site’s business practices (or by specific user behavior), best practices are often site specific. This is Capstorm’s recommendation:
If your site uses Salesforce quotes, then run a daily CopyStorm/Medic job to fix hard delete problems within the OpportunityLineItem table.
Each weekend, run CopyStorm/Medic to fix hard delete problems on all custom tables and core Salesforce tables. Core Salesforce tables are listed in the sample CopyStorm/Medic file found here.
In practice, we have found that very few custom or core Salesforce tables need to be scanned. Over time we reduce the number of tables scanned based on problems reported by CopyStorm/Medic.
There are a number of system tables in Salesforce which do not track deletions or allow restoration from the Salesforce recycle bin.
Tables that follow this behavior can be found in the CopyStorm database by running the following SQL query:
select distinct tableName from copyForceTableField where tableName not in (select tableName FROM copyForceTableField where name=’IsDeleted’);
Most of the time a site will not care if these tables contain deleted records. If your site requires that deleted data be pruned, the best practice is to periodically drop or truncate these tables. Once dropped, the next run of CopyStorm will automatically repopulate the tables.
High
When team assignments change hard deletes from Salesforce can happen.
.*History
Low
Older history records can be automatically hard deleted by Salesforce.
.*Share
Low
When sharing rules are modified, Saleforce hard deletes share records.
.*Feed
Low
Feed records (from Chatter) are sometimes deleted for reasons that are not clear. We suspect this occurs when a chatter feed is deleted.