Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Is possible to reduce the input buffer of a step?

  1. #1
    Join Date
    Jul 2007
    Posts
    8

    Default Is possible to reduce the input buffer of a step?

    I was wondering if it is possible to reduce the input buffer of a step?

    The background of this requirement is the following: I generate SQL queries dynamically from the data dictionary. Some of those - let's call them dimension tables - have only a couple of rows and take seconds to execute. Others - the fact tables - take hours.

    The queries themselves are executed biz a database lookup step that runs with 3 instances. Now it can happen is that the queries are distributed very unfortunately, e.g. that two steps receive only dimension queries and the third gets all fact tables. Once the dimension queries are done the two steps will shout down since the other fact tables are in the input buffer of the third step.

    So to solve that problem I would like all the queries to remain in the output buffer of the step that generates the and only to be forwarded once a step is idle to ensure all steps work as long in parallel as possible. Is that possible?

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    You can configure the Row Set size in the transformation settings dialog, the misc tab.
    Alternatively you can try to enable the blocking batching row set by setting the corresponding variable in the kettle properties file (use the editor).
    That one only keeps a few rows as a buffer.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.