Hitachi Vantara Pentaho Community Forums
Results 1 to 16 of 16

Thread: The filter can't send out all rows?

  1. #1

    Default The filter can't send out all rows?

    Hi guys,

    have you ever found filter can't send out all messages?

    the filter received 2126 rows, while one step followed the filter (dosomething a in my attached pic) received 125 rows, and the other (dosomething b) got 1000 row, and then kettle stopped here. All the following steps were stopped as well. The log displays all steps are running, but no rows added. everything stopped.

    I had three lookups in my transformation which you can found in the trf.

    NOTE: do something is actually something like sort, group by, that's, there's some calculation between lookup and the filter.

    what's the problem? is the problem of filter or lookup? I tested in both 2.5.0 and 3.0.0.

    if necessary, I can attached the whole transformation.
    Attached Images Attached Images  

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    Attach your transformation, but try to change it so that we can run it without requiring database access or so.... either by using row generators or by replacing datatabase input by text input file.

    Regards,
    Sven

  3. #3

    Default

    ok. Let me change it.

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    It won't fix your problem, but it allows us to have a look at it.

    Regards,
    Sven

  5. #5

    Default

    I made changes to the transformation, and I attached it as well as the source which contains some Chinese characters (anyway, I'm Chinese). Please made some minor changes to the Excel input as the path should not work in your computer.

    I replaced the lookup with the join as kettle 3.0.0 allows join (it doesn't work in kettle 2.50).

    Expecting your replies.
    Attached Files Attached Files

  6. #6

    Default

    more info...

    I found that joining/looking up two streams which was stream but divided into two by a filter could cause this problem.

    The transformation keeps running, but no more rows added to the stream. Definitely there are some problems here.

    Thanks,

    Samuel

  7. #7
    Join Date
    May 2006
    Posts
    4,882

    Default

    That's a known problem, if you have a good example make a tracker at http://jira.pentaho.org/browse/PDI

    Regards,
    Sven

  8. #8

    Default

    Thanks, Sven.

    I'll submit that to Jira later.

    But do you have any suggestions on how to avoid this step?

    I tried to add another source (the two same table input), which is too resource consuming and looks stupid to me.

    I also tried to use two filters to replace one filter ( I thought the problem of the filter at first), and it doesn;t work.
    Any more good solutions? I Really need it. my database now is really 2 busy. I can't do this with sql in database, but I have millions of projects should follow this way.

    God, please help me!!!!!!!!

  9. #9
    Join Date
    May 2006
    Posts
    4,882

    Default

    if you split a stream/hop you can never later join them together with a merge-join... so easiest is to duplicate your input. If your input is a table input step, use 2 table input steps extracting the same data, ...

    If you want to unload some burden off your database, extract the data once to a file and read that file twice e.g.

    God is no longer here

    Regards,
    Sven

  10. #10

    Default

    thanks, Sven.

    I'll make two inputs.

    This problem seems common to me (not sure with others), so can it be solved in 3.0 RC1?

    btw, God is always beside you if you believe Him .

  11. #11
    Join Date
    May 2006
    Posts
    4,882

    Default

    The original idea was to pop up a dialog telling you that you're doing something not allowed. We'll have to check... make a tracker ;-)

    Thanks,
    Sven

  12. #12
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Sven,

    I think this problem occurs if you only send rows over one arm and none over the other.
    Since the merging step reads from one input and then the other, it blocks on the input that doesn't receive anything.
    The buffers in the other arm fill up and then the transformation stalls.

    In 3.0, this specific problem will no longer occur.
    That is because the merging step will always try to deplete both input streams, no matter what.
    I had to fix this to solve a clustering/repartitioning problem.

    HTH,

    Matt

  13. #13
    Join Date
    May 2006
    Posts
    4,882

    Default

    Ok, didn't know that... I was only aware of the problem in 2.5.x. where there was suggested to put a warning/error dialog on that condition.

    Regards,
    Sven

  14. #14

    Default

    Thanks, Sven and Matt.

    is this solved in 3.0? M2 should still have the problem as I'm testing it on m2 of 3.0.

    I hope I can use only one input~.

  15. #15
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    M2 should do the trick. If there aren't any other problems at play here.
    Try these to update M2:

    http://s3.amazonaws.com/kettle3/kettle-engine-3.0.jar
    http://s3.amazonaws.com/kettle3/kettle-ui-swt-3.0.jar

    HTH,

    Matt

  16. #16

    Default

    Thanks, Matt.

    But how can I update M2? I downloaded the two files , then copied them into Kettle/lib and replaced the former ones.
    After that, I tried to start spoon. The interface appeared and disappeared immediately.

    Any method to update this? thanks.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.