Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Text file problem -- preview rows

  1. #1
    Join Date
    Jul 2007
    Posts
    4

    Default Text file problem -- preview rows

    I'm using 'text file input' and running into a problem. The file has 16 rows, with several thousand fields (not exactly sure how many)

    In the dialouge box I can add the file, show content, and view first data line, and I get back data results. When I try to preview rows I get this error...

    Any suggestions?

    Andy

    ---------

    2007/07/25 10:39:21 - org.pentaho.di.trans.Trans - This is not a replay transformation
    2007/07/25 10:39:21 - org.pentaho.di.trans.Trans - This transformation can be replayed with replay date: 2007/07/25 10:39:21
    2007/07/25 10:39:21 - org.pentaho.di.trans.Trans - Initialising 2 steps...
    2007/07/25 10:39:21 - Text file input.0 - Starting to run...
    2007/07/25 10:39:21 - Text file input.0 - Opening file: C:\Documents and Settings\Andrew\Desktop\Playground\cnty_try2.csv
    2007/07/25 10:39:21 - dummy.0 - Starting to run...
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : Unexpected error : org.pentaho.di.core.exception.KettleException:
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : Error converting line
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) :
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : Error converting line : java.lang.ArrayIndexOutOfBoundsException: 0
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : 0
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : org.pentaho.di.core.exception.KettleException:
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : Error converting line
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) :
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : Error converting line : java.lang.ArrayIndexOutOfBoundsException: 0
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : 0
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) :
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) :
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : at org.pentaho.di.trans.steps.textfileinput.TextFileInput.convertLineToRow(TextFileInput.java:742)
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : at org.pentaho.di.trans.steps.textfileinput.TextFileInput.processRow(TextFileInput.java:999)
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : at org.pentaho.di.trans.steps.textfileinput.TextFileInput.run(TextFileInput.java:1396)
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : Caused by: org.pentaho.di.core.exception.KettleException:
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : Error converting line : java.lang.ArrayIndexOutOfBoundsException: 0
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : 0
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) :
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : at org.pentaho.di.trans.steps.textfileinput.TextFileInput.convertLineToStrings(TextFileInput.java:592)
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : at org.pentaho.di.trans.steps.textfileinput.TextFileInput.convertLineToRow(TextFileInput.java:625)
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : ... 2 more
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : at org.pentaho.di.trans.steps.textfileinput.TextFileInput.convertLineToStrings(TextFileInput.java:551)
    2007/07/25 10:39:21 - Text file input.0 - ERROR (version 3.0.0-M1, build 300007 from 2007/07/19 16:05:15) : ... 3 more
    2007/07/25 10:39:21 - Text file input.0 - Finished processing (I=2, O=0, R=0, W=0, U=0, E=1)
    2007/07/25 10:39:21 - dummy.0 - Finished processing (I=0, O=0, R=0, W=0, U=0, E=0)

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    Raise a bug report at http://kettle.javaforge.com ... I would suggest to start the description with v3.
    And in the meantime use v2.5 ... v3.0 is not going be production ready for a while.

    Also let us know if it works with v2.5.

    Regards,
    Sven

  3. #3
    Join Date
    Jul 2007
    Posts
    4

    Default

    Thanks Sven,

    I switched to 2.5 and there is some improvement. When I click preview rows, I now get a window with the proper number of rows -- If I preview 500 I get 15.

    But there is no field data. When I click 'Get fields' the scroll bar shoots up the right side of the screen and then the program hangs up.

    I can open the file fine in Excel and Access, except they truncate the number of fields.

    So I know that Kettle is reading the line breaks properly, and I have the file is delimited by commas... is there a limit on the number of fields?

    I'll add the bug to javaforge.

  4. #4
    Join Date
    Jul 2007
    Posts
    4

    Default

    For clarification there are ~160 thousand fields and 16 rows (including header) in the file I'm trying to load.

  5. #5
    Join Date
    May 2006
    Posts
    4,882

    Default

    160.000 fields per row?

  6. #6
    Join Date
    Jul 2007
    Posts
    4

    Default

    Yeah, it's the US Census Burea's Decennial census.

    What is the limit on the number of fields? I might be able to get the data from someplace else.

  7. #7
    Join Date
    May 2006
    Posts
    4,882

    Default

    pffffffffffffff ... it will surely not display right in spoon... there's a limitation e.g. in Windows on how many columns there can be in a regular table (about 1500 maximum I think). The real processing will do more than 1500, but I doubt anyone ever tried 160K fields.

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.