Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Is this a bug - Hanging transform

  1. #1
    Join Date
    Apr 2007
    Posts
    2,010

    Default Is this a bug - Hanging transform

    Hi,

    Say you have a table input step. you want to use it as a source for 2 other steps.

    The first - a data validator. or some other stream lookup step.
    The second a merge join.

    Say you have a small rowset size. ( 500 )

    So; you run it for say 50k records. It hangs; Because the data validator needs to read ALL the rows from the table input step, but the table input step is blocked because it cannot write any more than 500 to the merge join step.

    Errrm, maybe I need a pic Anyway, is it a bug? or is it user error (Caused by over zealous performance tuning) ?

    Dan

  2. #2
    Join Date
    Dec 2009
    Posts
    609

    Default

    Dan,

    use the "Dummy" Step to combine two incoming streams... the merge join might cause a kind of deadlock situation

    ...hmm, after reading your question exactly I would try to use the Dummy step as first step after the "Table Input" and then broadcast your data to the two other steps from there....

    HTH,

    Tom

  3. #3
    Join Date
    Apr 2007
    Posts
    2,010

    Default

    using a dummy step like that only gives you more rowset cache, and it could potentially hide the issue, but fundamentally it still has the same problem if you have enough records.
    However, Matt may well have fixed this now, in the sense that there is now an exception in this scenario, see this great description here: http://jira.pentaho.com/browse/PDI-7414

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.