Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Exclude new columns from csv step

  1. #1
    Join Date
    Feb 2011
    Posts
    3

    Default Exclude new columns from csv step

    Hi, I have a question about the CSV/Text file input module.
    I have to import a CSV file with, for example, 3 fields: field1, field2, field3.
    I configure a text file input (or csv) step to read 3 columns. The fields name are the same as the headers.
    Now, I add a new column in the input file, between field1 and field2 (field1, newfield, field2, field3) but not in the step under the tab fields.
    If I do a preview data i have the field2 value shifted in the 3° column, and the field2 value is the new column value.

    Is it correct?
    Is there any possibility to exclude the new columns?

    Thanks.
    Last edited by ilfra77; 02-04-2011 at 11:54 AM.

  2. #2
    Join Date
    Nov 2008
    Posts
    143

    Default

    Consider the CSV input as a "file parser specification": If you tell it that there are only two columns, it'll read two collumns.

    If you want it to ignore other columns, use a "Select values" and in the first tab, put only the columns you want to keep in the stream.

  3. #3
    Join Date
    Feb 2011
    Posts
    3

    Default

    Quote Originally Posted by renatopb View Post
    Consider the CSV input as a "file parser specification": If you tell it that there are only two columns, it'll read two collumns.
    Ok, but after I add a new column between the 2 original column in the csv file, the step read the first 2 column. It should read column 1 and 3.

  4. #4
    Join Date
    Nov 2008
    Posts
    143

    Default

    Not really.
    The parser reads a line, then splits it in each comma, making an array.
    After that, it looks at your row metadata and checks how many columns you have specified.
    Since you only told it you had 2 columns, it'll read the first two objects in the array and passes it on, ignoring the other elements inside the array.
    Got it?

  5. #5
    Join Date
    Feb 2011
    Posts
    3

    Default

    Correct!
    And it is exactly what I feared: if the input csv file changes (for instance columns added) the step doesn't work anymore. Except specify other columns.

  6. #6
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    Hi ilfra.
    Yes, if the csv changes then you have to update your Text File Input step as well.
    What you could do is to create a previous step witha condition, pointing to 2 different Text File Input Steps.
    If a file has 4 columns, then choose Text File Input 1, otherwise choose Text File Input 2.

    Mick
    PS: you can private post me in Italian if needed

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.