Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Merging two streams the number of fields in output is doubled

  1. #1

    Default Merging two streams the number of fields in output is doubled

    Hi,
    I need to merge two same structured streams (from the same table but with two different select and group_concat command).

    The output has to be the sum of all lines of one stream to the rows of other one.
    I used FULL OUTER merge.

    I.e. 3 rows from 1st and 1 rows of 2nd. Output result 4 rows.

    Why the output stream make a kind of copy of all fields adding _1 ???

    The output is at the end of 3 rows

    Thanks
    Attached Images Attached Images  

  2. #2

    Default

    OK probably I did a mistake using Merge Join instead of Sorted Merge.
    Is that one the problem ?

  3. #3
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    You can use "Append streams", "Sorted Merge" or no step at all to unite two input streams.
    The subtle differences should be obvious after reading the step documentation.
    So long, and thanks for all the fish.

  4. #4
    Join Date
    Feb 2011
    Posts
    840

    Default

    marabu got it almost perfectly =p but yeah, reading docs would solve the problem.

    Main difference is that Append Streams is mostly cosmetic and useless, while Sorted Merge needs sorted inputs and it's output is sorted.

    FULL OUTER is a merge join... actually, now that I think about it, this part of PDI can really be awesomely confusing for newcomers...
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  5. #5
    Join Date
    Mar 2011
    Posts
    19

    Default

    Second use of append streams, or just join them up if the structure and order is identical.

    I see Append as synonymous with a UNION, rather than the merges as more of a join when I use them.

    subtle differences should be obvious after reading the step documentation
    <- this guy's funny although I admit the doco's getting better, it's the subtleties which tend to be unclear.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.