Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Data not syncronize, transformation running long

  1. #1
    Join Date
    Dec 2016
    Posts
    17

    Unhappy Data not syncronize, transformation running long

    Hello all,
    I have two input stream, one from db and other is file.
    For db i used table input step and for other i use file input step. After that i use stream lookup step in order to see that records exist in db or not , but transformation running long, as still data is getting fetched from db, and stream lookup start work as data read from file is very fast.
    Can anyone provide me solutions on this, also i do not want to sort the data, and i want all records from table as well for further processing.

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    In your design you had to make a decision between Lookup and Join.
    Can you tell us what exactly made you select Lookup?

  3. #3
    Join Date
    Dec 2016
    Posts
    17

    Default

    Thanks marabu for ur response.
    I select streamlookup because i do not want to sort data unnecessary, also as per current requirements i just want to check records exist or not and not data sorted.

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    You have chosen Stream-Lookup because you don't have to sort the rows?
    Stream-Lookup will hash the key values, so lookup will rely on binary search followed by sequential search in case of collisions, I think.
    If your decision against Merge-Join isn't the result of a careful analysis, try Merge-Join already.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.