Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Parallel execution of dynamic number of transforms?

  1. #1
    Join Date
    Aug 2016
    Posts
    11

    Default Parallel execution of dynamic number of transforms?

    Hey guys.


    I currently have a daily PDI routine set up which dumps a few of our tables into CSV files, then uploads them all to Amazon Redshift.


    We do this for about 15 tables right now. Which tables get dumped is determined by an auxiliary table with a table_name column as well as some metadata like row limit, ordering key etc.


    So. Right now, I run all the CSV dumps in parallel, by calling separate, parametrized instances of kettle. This was always meant to be a placeholder solution since I knew that I'd be running out of memory eventually.


    I know I can run them all from within the same instance of kettle by reading the table list, setting a parameter, executing once for each row etc. Running them serially would take too long, however, so I'd like to keep the parallel execution.


    I thought of writing an external script to straight up edit the kjb file before execution based on the table list but that seems like a very crude solution. Are there any other ways to accomplish this from within PDI?


    cheers,

  2. #2
    Join Date
    Aug 2011
    Posts
    360

    Default

    You can use the job/trans executor inside a transformation with multiple copies defined by a parameter
    of the transformation.
    Do a trans with a paramater nbrOfJob.
    Then table input (get list of your tables) -> job executor: execute for each row, use fields as parameter of job,
    set multiple copies of the step and put ${nbrOfJob} as number of copies.

    Then you can launch your transformation choosing number of jobs in parallel.
    We do that and we compute number of jobs in function of the number of processor of each
    environnment.

  3. #3
    Join Date
    Aug 2016
    Posts
    11

    Default

    Worked like a charm. Thanks!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.