Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Feature suggestion: load balancing parallel steps

  1. #1
    Join Date
    Nov 2010
    Posts
    9

    Default Feature suggestion: load balancing parallel steps

    At this moment (pdi 4.2), the output of a task can be chosen to be "copy" or "distribute" I would like to have a third option "balance". That option would put the available output row onto the input row of a step that would not block the insertion of the data into its input queue.
    The current option "distribute will make the transformation "as slow as the slowest step".
    The proposed option "balance" will make the transformation "as fast as the fastest step".

    Scenario:
    I have a few servers that all do the same thing, but differ in performance. The performance can vary in time, so a fixed setup is not a good solution.

    Name:  pentaho-load-balancer.jpg
Views: 126
Size:  8.4 KB

    I already have created a UDJC transformation step that just does this (load balancing, see attachment), but wouldn't it be nicer to have it as an extra output-option?

    Carl van Denzen
    Attached Files Attached Files

  2. #2
    Join Date
    Nov 1999
    Posts
    459

    Default

    Quick FYI: Have a look at
    http://jira.pentaho.com/browse/PDI-7924
    As an ETL developer I want to have built-in support for load balancing in Kettle transformations

    Thanks & best regards,
    Jens

  3. #3
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    What Jens means is that it's a standard feature of 5.0.


  4. #4

    Default

    I'd just like to add that this is awesome. Really looking forward to this!

    Keep up the good work guys!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.