How to Solve OPERATION_TOO_LARGE Errors

If CopyStorm is reporting an OPERATION_TOO_LARGE error, it will be included in the CopyStorm log. The error will look like this:

09:02:11:  [UnexpectedErrorFault [ApiFault exceptionCode=’OPERATION_TOO_LARGE’ exceptionMessage=’exceeded 20000 distinct ids’ extendedErrorDetails='{[0]}’ ] ] Before diving into the root cause, this is the most common solution:

• Grant the ViewAllData privilege to the Salesforce credentials being used with CopyStorm.

The root cause of this exception can be found in the Salesforce description of the error:

OPERATION_TOO_LARGE

The query has returned too many results. If certain queries are run by a user without the “View All Data” permission and many records are returned, the queries require sharing rule checking. For example, consider queries that are run on objects, such as Task, that use a polymorphic foreign key. These queries return this exception because the operation requires too many resources. To correct, add filters to the query to narrow the scope, or use filters such as date ranges to break the query up into a series of smaller queries.

You can find the Salesforce documentation in the Core Data Types section.

Almost all CopyStorm sites will want to use Salesforce credentials with the ViewAllData permission because without this permission all data will not be backed up. In particular, no metadata will be backed up because the API call used to fetch metadata requires the ViewAllData permission.

If your organization does not want to give your CopyStorm backup user the ViewAllData permission and  an incomplete backup is OK, you can configure CopyStorm to backup data using a sliding timebox.

For example, to backup the Note table using a 60 day sliding timebox, the following TableRule must be added to your site’s TableRuleRegistry.xml file:

This XML rule means:

• Use a sliding 60 day window when backing up Note.
• Start the first search on 1-Jan-2001.
• Start subsequent searches at the timestamp on the most recent Note record in the backup.

The system will backup Note records in roughly the following order:

1-Jan-2001 thru 28-Feb-2001
1-Mar-2001 thru 30-Apr-2001
1-May-2001 thru 30-Jun-2001
etc… until the current date is reached.