Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Using Metadata in a transformation

  1. #1
    Join Date
    Jun 2014
    Posts
    28

    Default Using Metadata in a transformation

    I know I can get column/stream names using the Utility > Metadata structure of a stream step.

    Where I'm having a conceptual failure is how I can use that information.

    I want to be able to define certain columns as 'single response' or 'text' in one file.

    Then use that information on my main input file and process the columns I have identified as 'single response' in one sub-transformation and those I've identified as 'text' in a separate transformation.

    I have multiple files in the same format but each has their own processing needs based on the column/field name.

    For example all the files have FIELDA, FIELDB, FIELDC - but in some of the files FIELDA is 'single response' and in others it's 'text'.

    I want to be able to process each file with the same transformation in conjunction with a controlling file that will identify FIELDA as 'single response' or 'text' for that particular file.
    I'm having trouble visualizing how I can tell tell Kettle how to process based on the field name.

    Any help would be appreciated.

  2. #2
    Join Date
    Aug 2011
    Posts
    360

    Default

    Maybe:
    first read only the header line of the files, so no header+limit 1 row, such that you
    Have the name of fields in the rows for each file.
    Do the logic to route each filename depending on fields names (or add a flag identifying the type of file)
    Then read the full content of the files, without header, headerso you now have a flag on each row to route
    The data where it should go

  3. #3
    Join Date
    Jun 2014
    Posts
    28

    Default

    Thanks Mathias, I think that will work. I doubt I'd have thought of reading the file twice, once for just the header and again for the data (obviously I hadn't) - but that's exactly the kind of conceptual help I needed. Thanks again.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.