Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: How can I execute th jobs in paralell ?

  1. #1

    Default How can I execute th jobs in paralell ?

    Attachment: screen.JPG Dear All


    As I read in doc the transformations run sequencially.
    Is there any possibilities run it paralell ?



    Regards
    G

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default RE: How can I execute th jobs in paralell ?

    No.

  3. #3
    Join Date
    May 2006
    Posts
    29

    Default RE: How can I execute th jobs in paralell ?

    Well, Couldn't you? I mean, maybe not from within one model, but can't you have the OS schedule execution in parallel?

    The hard part is to synchronize afterwards.

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default RE: How can I execute th jobs in paralell ?

    Actually, you could. But for now you can't.

    And it would need quite a bit more logic than simply to send it off in parallel. That's the easy part.
    If you would want to do it correctly, you would have to take into account all kinds of dependencies.

    Originally I had a "run in parallel flag" in the GUI, but removed it because the complexities would be too great.

    Perhaps we can tackly this one of the next versions.

  5. #5
    Join Date
    Jan 2006
    Posts
    14

    Default RE: How can I execute th jobs in paralell ?

    As Matt noted, the dependency issues would be too great to easily and reliably handle for the general case among multiple transformations.

    My own Kettle usage has mainly been to quickly address simple data migration tasks rather than complex data warehouse transformations. One thing Kettle *does* do is, if a single transformation consists of multiple read/write steps each joined by a single hop (e.g. multiple table copies, each independent of each other), try to run them in parallel. This is a special case where (a) everything is in a single transformation, and (b) the connected hops connect distinct tables, e.g. readTableA -> writeTableA, readTableB -> writeTableB within a single transformation would result in two threads so the tables are effectively copied in parallel.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.