Working off mass data in the process

Issue

A large number of datasets (>10,000) are to be created in a server-side process script. Each dataset is generated via a call-up of the method WorkSpaceScriptUtils.createEntry() to ensure that every server logic is applied to the new datasets at the time of creation. This method, however, is getting slower with the increasing number of newly created or changed datasets. This problem originates in the transaction management of the application server.

The following script now attempts to create 100,000 activities. The transaction performance does not allow a process completion within 24 hours, which results in the execution being terminated with errors.

Creating 100,000 activities in a script

IScriptWorkSpace acWs = WorkSpaceScriptUtils.createEmptyWorkSpace("Activity")
for(int i = 0; i < 100000; i++) {
  String subject = "Akt100000 " + i
  IContainer ac = WorkSpaceScriptUtils.createDefaultEntry("Activity");
  WorkSpaceScriptUtils.setValue(ac, "Subject.Activity", subject)
  WorkSpaceScriptUtils.setValue(ac, "StartDate.Activity", DateUtils.NOW)
  ...

  WorkSpaceScriptUtils.createEntry(acWs, 0, ac)
}

Process loop and transaction interruption

One frequently used solution approach is to sub divide the creation process into e.g. 100 packets with 1000 datasets each.

This solution will always only use a smaller IScriptWorkSpace, but will stay within one transaction. The process execution will also not be interrupted in the loop, which will result in a StackOverflowError in JVM!

The transaction must have interruptions for each new entity for each package to run at the same performance level. Generally, mass data processes should always be based on an asynchronous process execution, because the process caller will not want to actively wait for the process end. The right solution will interrupt the loop with a Transaction intermediate event:

The transaction interruption ensures a consistently high performance and error-free execution of the new creation and informs the caller about the result at the end of the process. This approach allows the processing of large amounts of data.

It is, however, not the right approach for the quick processing of mass data that need no other server logic. For uninterrupted processing of mass data with individual transactions, the mass data transactions, outsourced on the mass data server, should be used in the process.