Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: How can process multiple records with every record in indenpendent transactions

  1. #1
    Join Date
    Jun 2016
    Posts
    7

    Default How can process multiple records with every record in indenpendent transactions

    Hi,

    We are now using flow to process message, for every message we will validates the values and update multiple tables, so we need to put all the steps into one transaction. Every message need its database transaction, so we cannot process multiple messages at one time, we now execute flow in java codes, for every message, we need to reinitialize all the flows, but as we contains many steps, it take a long time to initialization all the steps, when we process messages one by one, it will become more longer, anyone know how to speed up it?

    Either we can speed up the initialization time, or we can process multiple messages one time(make sure the good message can commit, bad message will rollback), for the 2nd solution, I'm now thinking of creating a custom plugin, the plugin save all the added/updated/delted db records into memory, at the end of the flow, we check if the message is okay, we commit, if not okay, we don't commit.
    So is there any good solution we can use?

  2. #2
    Join Date
    Jun 2016
    Posts
    7

    Default

    I have found a solution to solve this issue, we use listeners to get the sql script and parameters from all the DB steps and rollback the data in the final PDI flow step, then we recommit all the data in the java codes.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.