Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: processing two input files

  1. #1
    Join Date
    Sep 2017
    Posts
    27

    Default processing two input files

    I'm processing two input files by using either text input or fixed file in PDI. I would need to compare the fields like social security and phone numbers in both files. If match found, write to one output and if match not found, write to a different output file. Going into Transformation, which step would match such fields on both files. Is there a working example on website that does something like that?

    Thanks,

  2. #2
    Join Date
    May 2017
    Posts
    10

    Default

    See below thread ,your requirement is most likely to this thread.


    Thread: Help filtering rows from a list of values

  3. #3
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    A Full Outer Merge-Join would be my first choice here.


    Name:  230452.png
Views: 63
Size:  15.7 KB
    So long, and thanks for all the fish.

  4. #4
    Join Date
    Sep 2017
    Posts
    27

    Default

    Thank you for this demo. I tried this Merge Join process. It works for me. I also tried Stream Lookup so that I can compare how it works in both ways. Now, question on the Stream Lookup - When I edited the Stream Value Lookup to fill out the source fields and the Lookup fields for matching values. I assumed that if a match found, it would continue processing data to next steps but it isn't the case. Even if the data from the source file does not match the ones from the Lookup file, it still processes to next steps in the transformation. (I thought it would drop records if match is not found). I'm assuming I would need to filter out unneeded data after the Lookup step?

    Thank you

  5. #5
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Stream-Lookup conditionally adds the specified fields to each row, much like a Left-Outer-Join would do.
    So, yes, you'll have to filter rows.

  6. #6
    Join Date
    Sep 2017
    Posts
    27

    Default

    Alright. Thank you for your help.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.