Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Handle text file input with either comma separator or tab separator

  1. #1
    Join Date
    Jan 2018
    Posts
    2

    Exclamation Handle text file input with either comma separator or tab separator

    Hello there,

    I need to handle input files with .txt extension which can be either comma-separated or tab-separated. My questions are

    1. Is there any easy way to extract the separator? I am thinking to extract the separator and set it to a variable.

    OR

    2. Can I specify multiple separators in the content page of Text File Input step? If not, is there any way to handle this?

    Would appreciate if any response!

    Thanks!

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    See, there's a reason why we have data contracts in the first place.

    I doubt there isn't a way to tell from the file's metadata what field separator is used.
    Usually, you don't have two formats coming from the same source.

    Anyway, you can extract the field separator from the first line of text.
    Just limit processing to a single row in Text-File-Input.
    Read the whole line as a single field.
    You can use Filter-Rows to detect the field separator, then, and set a variable to be used in a different transformation.
    So long, and thanks for all the fish.

  3. #3
    Join Date
    Jan 2018
    Posts
    2

    Default

    Thanks for your prompt response! Could you tell me how to read a whole line as a single field? Where to define the field? I didn't see it within the Text File Input step.

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Choose FIXED format and set up a single field named e.g. line starting at offset 0 with length 9999 trimmed on the right side.

  5. #5

    Default

    Hi

    I want to handle same scenario.
    But my question is, using filter rows step, how can we set same variable twice?
    I mean if separator is comma then it will set variable to comma but execution will go in other direction too and again variable will be set to tab.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.