Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Run Job entry in Parallel for multiple result rows

  1. #1
    Join Date
    Oct 2012
    Posts
    11

    Default Run Job entry in Parallel for multiple result rows

    Hi All,

    I have a job, JOB1 ( Trans1 -> JOB2). In Trans1 i set some rows to result. I want to run JOB2 for each row so i have checked "Execute for each input row". JOB2 runs serially for each row. So in case there are many rows job will launch for next row only when first row finishes. Wanted to know if there is some way to run JOB2 for result rows in parallel.

    Thanks

  2. #2
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    I guess that you have to create multiple jobs using different input rows :-/
    -- Mick --

  3. #3
    Join Date
    Oct 2012
    Posts
    11

    Default

    Hello Mick,

    Could you please elaborate. Say for eg i have 2 rows added to result in transformation 1. Do i have to add 2 job entries pointing to JOB2? In that case, how do i connect the Trans1 and JOB2 such that rows get distributed across the entries instead of both rows going to both entries.

    Thanks

  4. #4
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    I would do something more SIMPLE (and probably too basic for your needs):
    create 2 (or more) identical jobs and then simply change the input file so that each job import only some rows from the same input file.
    For example, job 1 select rows from 1 to 2, job 2 get rows from 3 to 5.. and so on.
    Not very clever, I know, but I can't think of a different way.
    -- Mick --

  5. #5
    Join Date
    Jan 2014
    Posts
    25

    Default

    In Pentaho 5 you have the step called Job Executor in transformations. This step can be the last step in Trans1.
    The 'Change number of copies to start' of the Job Executor step can be increased for parallelism.

    If you are not using Pentaho 5 you can use the User Defined Java Class to invoke a job with the required parameters.

  6. #6
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    And... the number of copies of a step can be set as a variable in version 5!

  7. #7

    Default

    Any solution Tathagat?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.