Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Parameter driven Parallel Execution of Jobs/Transformations

  1. #1
    Join Date
    Oct 2011
    Posts
    12

    Default Parameter driven Parallel Execution of Jobs/Transformations

    Hi there,

    how can i start parallel instances of a job/transformation dynamically?

    I have to load some bigger tables and i want to set the number of instances working parallel to load the table as a parameter.

    The job/transformation uses the instance-parameters to load the table using Modulo.
    iE. SELECT ... FROM TABLE WHERE MOD(table_id, ${Instances} ) = ${InstanceCount}

    Any idea? Or is there an other way for my performance issue?

    The picture shows the static version.

    Thanks...
    Barni
    Attached Images Attached Images  

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    In 4.3.0-M1 we have created the Job Executor step for this purpose.
    However, I don't know if the internal variables from the step are passed down to the executing job, I'll have to check that.

    The alternative is to create a little framework with a Carte instance where you fire load jobs at. That shouldn't be too hard to do either.
    It can be as simple or as complex as you want (for example: http://diethardsteiner.blogspot.com/...designing.html)

  3. #3
    Join Date
    Oct 2011
    Posts
    12

    Default

    Thank you Matt for your fast response.

    The Job Executor sounds perfekt for that. But when will the 4.3.0 be available?

    I will take a closer look at your "highly available scalable solution" first to get an idea of writing my own little framework.

    Barni

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    4.3.0 is scheduled for now to be ready around April 2012.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.