Help! Partition Creation Failed

When partitioning tables containing large volumes of data or Base64-encoded content, the partition creation process can fail mid-way due to various issues, such as:

  • Application freezes or crashes
  • Transient network errors
  • Resource exhaustion (e.g. memory or CPU spikes)
  • Timeouts or I/O bottlenecks

These interruptions can leave partitions in an incomplete or inconsistent state.

The Partition Creation process is designed to be incremental:

  • Records are copied from Source to Target Partition in an ID-based ordering (lexicographical).
  • After each batch of records is copied from Source to Target, the last read Salesforce ID is persisted in the Database.
    • See the CopyForceTablePartitionLoadTask database table.

Note: CopyStorm will refuse to replicate data to a table if it has a Partition Creation In-Progress

If partition creation fails, you can resume the process by navigating back to the Partition Editor Tool. You’ll see the following dialog appear if a partition creation is in-progress:

From this point, you’ll have the option to:

  • Resume the Partition Creation process
    • This will start copying data from Source to Target from the Last Read Salesforce ID.
  • Cancel the Partition Creation process
    • Note: Cancelling will not remove data already copied if the Source is S3 or a Folder
    • Cancelling a JDBC Partition Creation will delete the new JDBC Partition.
    • Source data will never be deleted during the cancellation process.