Hitachi Vantara Pentaho Community Forums
Results 1 to 8 of 8

Thread: Variable # of Columns in CSV File

  1. #1

    Default Variable # of Columns in CSV File

    Hi,

    Kettle is badass and stuff.

    I ran into something I couldn't figure out with a strangely formatted delimited text file.

    I know I can probably accomodate this with a javascript and/or something outside of spoon, but in my (mostly futile) attempt to keep everything flat-file based and not become a javascript programmer, I'm wondering if it can be handled in text input.

    The file is delimited with a header row, but the header doesn't represent all the columns present in the file. While the columns that are present are consistent throughout the file, the number of columns for each row is not consistent.

    example:

    header1, header2
    field1, field2, field3
    field1, field2, field3, field4

    In the layout screen, if I define the 'max' number of fields it errors out and says it can't parse null strings. If I define what's in the header I get the ArrayIndexOutOfBoundsException.

    If this can't be done in the text input - no worries, but just wanted to make sure there wasn't a known way to handle files like this. Any feedback is appreciated.

    Thanks,
    -Drew

  2. #2

    Default

    What version of Kettle?

    I'm using 2.5.0, and do the number of fields = max number of fields approach and have not run into these issues with Text Input, even if not all the fields exist in a CSV file.

    Are these errors from the Text Input or a different step?

  3. #3
    Join Date
    May 2006
    Posts
    4,882

    Default

    Darren,

    Can you download 3.0.0-M2 and test your own transformations for the case of the original poster

    Regards,
    Sven

  4. #4

    Default

    Sorry, forgot to include version -- 3.0.0M2.

    I did as you suggested and ran in 2.5.1 - that did it. It doesn't error out and the max columns approach works fine.

    Thanks for looking into this,
    -Drew

  5. #5

    Default

    Good catch, I'll need to be careful with a couple of jobs when migrating to 3.0 ;-)

  6. #6
    Join Date
    May 2006
    Posts
    4,882

    Default

    Maybe post a tracker with a small example transformation?

    Regards,
    Sven

  7. #7

    Default

    Added tracker, no examples in tracker but pretty self-explanatory in forum thread (almost forgot about this, creating tracker before I forget again!):

    http://jira.pentaho.org/browse/PDI-199

  8. #8
    Join Date
    May 2006
    Posts
    4,882

    Default

    I really would try to add small examples if you can... it helps the people fixing the bugs . The clearer your problem the earlier it will be fixed.

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.