CopyStorm 7.37.10, 7.34.10

Release Date: 3-Aug-2016

This is a minor feature and bug fix release.

Ignore Salesforce Field Width Changes that would Shorten a Column in the Database

Though Salesforce permits a string field to become shorter, this is not a good idea for a relational database. Salesforce will allow a shortened string column to contain data that no longer meets the maximum length limitation — relational database will not.

Support Overriding Proxy Parameters on the Command LIne

Previous versions required that all proxy parameters be hard-coded into a configuration file. This release supports overriding the proxy parameters in a configuration file values from command line switches. See this page for the new command line switches.

Improve the Discovery of Records to Delete Performance

An unnecessary ORDER BY clause was removed from the SQL statement used to find records that should be deleted.

Improve Performance when Just the Schema is Being Built

If the “Modified Since” CopyStorm parameter is set to a year in the future (example: 2100-01-01) CopyStorm will no longer query Salesforce for potential records to backup. This change was made to improved application performance when a user only wanted the Salesforce schema (and no data).

Catch an Obscure Salesforce Timestamp Error

When Salesforce instances are having problems, they sometimes return a null timestamp for a record. CopyStorm now detects this mistake and cleanly halts processing for the problem table.  Previously CopyStorm will report an exception and attempt to continue processing records from the table. More than likely your site has never seen this error.

Automatically Repair a MySQL CREATE TABLE Statment when the Table is Too Wide

If a MySQL table is declared wider than the database allows CopyStorm now converts the widest VARCHAR columns to TEXT types until the table is narrow enough to work. Previously the only recourse was for a use to adjust the “Max Varchars” parameter until the table was narrow enough.

Recover from Extremely Large Attachments and ContentVersion Objects

When backing up extremely large Attachments and ContentVersion records (>800MB) and unzip error can occur when parsing the response from Salesforce. It is unclear whether the error is function of an incomplete message from Salesforce or a problem with the core GZipInputStream in Java (forums indicate that it may be a problem in Java libraries but inspection of the SOAP enveloper indicate a Salesforce issue). This version of CopyStorm will recover from this error and continue to the next table. Previous version would attempt to read a problem record N times and then stop completely.

Log an Exception When CopyStorm is Forced to Reconnect to Salesforce

This extra bit of logging will help diagnosis odd Salesforce connectivity issues. It does not effect the function of the CopyStorm.

Make an Ignore Timestamps Backup Restartable

The Ignore Timestamps does a backup based on the natural order of Salesforce record ids. Though rarely useful, it is now restartable if interrupted. The type of backup is used only when Salesforce timeouts prevent the initial load of a table from succeeding — typically only for organizations with extremely large numbers of historical tasks.

Optimize Parameter Display When Target Database is “Directory”

When the target database is really a directory of CSV files then hide the advanced options that do not apply.

Add a Logging Table for DDL Operations

CopyStorm now logs DDL operations in the table CopyForceDDLLog. The primary purpose of this log to help Capstorm support look at history.

Address a Rare Issue with Unreliable and Slow Database Connections

If the connection to the CopyStorm database was slow and unreliable AND the connection to Salesforce was fast, there appears to be a condition then:

  • If all Salesforce reader threads were paused (because the reading data from Salesforce process was too far ahead of the database writing process).
  • If the database writer thread failed because the connection was dropped unexpectedly and became  incapable of being restablished in a a couple of minutes.
  • Then CopyStorm could become permanently paused.

We have seen this issue at a single site. At this site the database was throttled heavily from the cloud vendor and strictly limited IOPs while the connection to Salesforce was lightning fast.

Force CopyStorm to Stop When It Fails to Complete a Backup in a Specific Amount of Time

By default, CopyStorm will not stop running until a backup is complete or a user explicitly cancels the running job. This version adds a new advanced parameter, Maximum Runtime (expressed in minutes), that can be used to limit the total amount of time CopyStorm will use. If CopyStorm runs for longer than the specified Maximum Runtime:

  • CopyStorm will attempt an orderly shutdown by cancelling the job.
  • If CopyStorm fails to cancel in an orderly fashion within five minutes, then CopyStorm will forcefully exit.
  • When running in a batch mode, CopyStorm will always exit after the Maximum Runtime. In GUI mode, CopyStorm still will display a message indicating that the Maximum Rumtime was exceeded.

Limit the Amount of Time CopyStorm Spends on Specific Tables

A new parameter, maxRuntime, on a TableRule entry in the TableRuleRegistry.xml file can control the maximum number fo seconds CopyStorm will spend processing data from a specific table.  See the Table Rule Regisitry documentation.

Fix an Oracle Only Bug When Long String Column Sizes Change

In previous versions CopyStorm may attempt to ALTER a NCLOB column to be a CLOB column when the size of the corresponding field in Salesforce changes.

Make Oracle Much Smarter in Detecting Unsupported ALTER Operations

This release detects and provides better reporting on ALTER operations not supported by Oracle. Example:

  • When an ALTER would reduce a column to a VARCHAR when it is currently a CLOB, the ALTER request is silently ignored.
  • When an ALTER woudl convert a VARCHAR to a CLOB an exception telling the user what to do is returned (Oracle does not support this operation).
  • Old style NVARCHAR column types are always preserved on an ALTER. New columns, by default, are always VARCHAR2 when the database supports the UTF-8 character set.

Add a Flag to Capture a Complete Trace for All Messages To/From Salesforce

This new command line flag, -apiTraceFile <filename>, will cause all communication to and from Salesforce to be captured in a file.  We have found this feature useful in isolating bugs in the Salesforce infrastructure and decreasing the amount of time it takes for support to take interest in a case.

Fix a Bug When Transactions Were Limited

When the new Optimize Timestamps option was enabled, a fixed number of Salesforce transactions was allowed, and the application ran out of transactions then CopyStorm would record an invalide timestamp in its tracking tables and miss records the next time it ran. This bug only effects 7.37.9 versions that have these specific options selected.