Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: suggestion on running multiple parallel jobs

  1. #1
    Join Date
    Apr 2009
    Posts
    337

    Default suggestion on running multiple parallel jobs

    need some suggestion friends... I have to run multiple kettle jobs in parallel through an external scheduler, so I am planning to run them by calling the jobs from kitchen, now to my queries:
    • I am connecting to an enterprise repository to run these jobs.. is there a problem in that?
    • I have certain jobs to run in parallel, should I have a master job to carry out this action, or run multiple instances of kitchen, (all these jobs just dump data into certain staging tables, belonging to the same DB)
    • All these jobs are also updating certain audit tables and there is a risk of simultaneous access.. will there be any database locking issues, (transactional mode is enabled by default and database is postgres)
    Regards,
    Madhu

  2. #2
    Join Date
    Nov 2005
    Posts
    164

    Default

    Connecting to an EE repository should be fine, as the command takes a repository name. As for the transactional nature of the ETL, please note that transactional mode is only available in transformations. It is not supported among sub transformations or among jobs. Please see http://jira.pentaho.com/browse/PDI-977 and the related jira cases.

    - Brian Hagan

  3. #3
    Join Date
    Apr 2009
    Posts
    337

    Default

    Thanks Brian, The updation of audit table is done from within a transformation and not from the job, so I think that should be ok too then!
    Regards,
    Madhu

  4. #4
    Join Date
    Jun 2012
    Posts
    2

    Default How did you run multiple jobs in parallel?

    Quote Originally Posted by madhupenta View Post
    Thanks Brian, The updation of audit table is done from within a transformation and not from the job, so I think that should be ok too then!
    I have certain jobs to run in parallel, should I have a master job to carry out this action, or run multiple instances of kitchen, (all these jobs just dump data into certain staging tables, belonging to the same DB) - How were u able to do this? I tried this by scheduling 2 jobs at the same time but they ran in sequence and not in parallel!!
    Tried using InnoDB+autoinrement option in MySQL, creating 2 schedules for 2 jobs, attaching both the jobs to a single schedule
    , AUTOINCREMENT_SQL_FOR_BATCH_ID but nothing seems to be working. Still the jobs are getting executing in sequence

    I can make them work in parallel by creating a parent job and then triggering them in parallel within but i do not want to do it.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.