Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Record merge & flatten output

  1. #1

    Exclamation Record merge & flatten output

    Hello,

    I have to process data from a hardware vendor. Test data is captured for parts and errors are logged into a file ; parts can have one or many errors in one or many `error types`.

    There are several `errors type` fields which I am required to parse and retrieve a description for each error e.g

    Code:
    "PartIdNbr"|"COS_Errors"|"FUNC_Errors"
    21344667|3230;244MT|79GG
    My lookup process successfully parses each field and retrieves description like so

    Code:
    "COS_Errors"|"COS_Errors_Description"
    3230;244MT|Missing clip; Bad LED
    In my transformation, I have five streams for each error field. I want to merge all five streams into one target table but I can't seem to obtain all records on one single line. Instead, they end up `laddered` like so

    Code:
    "PartNbr"|"COS_Errors"|"COS_Errors_Description"|"FUNC_Errors"|"FUNC_Errors_Description"
    21344667|3230;244MT|Missing clip; Bad LED|Null|Null
    21344667|Null|Null|79GG|Incorrect software version
    Is there a way to flatten my output like this
    Code:
    "PartNbr"|"COS_Errors"|"COS_Errors_Description"|"FUNC_Errors"|"FUNC_Errors_Description"
    21344667|3230;244MT|Missing clip; Bad LED|79GG|Incorrect software version

    Thanks in advance for your input.

    Kettle v4.0 transformation attached. Attachment 6266
    Pentaho Data Integration CE 5.3.0.x
    JDK 1.7
    OS X Yosemite version 10.10.x
    MySQL 5.5.37
    Amazon Redshift
    Pacific Standard Time

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Perhaps add a "sorted merge" step to keep the records together by part nbr.
    Then perhaps use a "Group by" step on part nbr and use an aggregation on the fields. (first non null, etc)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.