Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Load handling in Kettle

  1. #1

    Question Load handling in Kettle

    Hi there

    I am wondering the size of data that can be handled by Kettle jobs. Is there a maximum limit to it on 2.3?

    I have a CSV file with 600,000 rows (70 MB size) and I use this data to load into a MySQL database. The Kettle job is running fine untill it loads the first 6000 rows after which it breaks. Kettle throws the error 'MySQL Out of range column adjusted, data trununcation..'. (I use MySQL as the database for kettle jobs)

    My questions are -

    1) Am I exceeding the load limit that is handled by Kettle jobs?
    2) Is there a work around available? (I could think of splitting the File Input / Increasing the memory size etc)

    I read some of the posts on the MySQL error there were not that useful..any pointers would be helpful.

    Thanks for the help!

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    There is no limit on the number of records loaded.

    You probably have an error in your input data. Put logging to rowlevel, start the transformation, go for a coffee, and afterwards check the row on which the transformation fails. Probably some field contains data longer than the column size in the table.

    Regards,
    Sven

  3. #3

    Default Data truncation issue still persistent

    Sven,

    I tried both of your suggestions.

    Even though I have increased the memory size to 1024 MB in chef.bat and increased the column width in the table (Int to BigInt / Varchar 1024 to 2048), the error is still existing.

    Please see the attachment for Kettle log.

    It seems to me that because of the data size (around 79 MB of CSV file), the error is thrown.

    Are there any way to increase the memory size or other ways to resolve the problem?

    Thanks to you!
    Attached Images Attached Images  

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    And I keep telling you it has nothing to do with memory

    Switch batching off, run again... the problem is already in your first row with NUMBER_OF_SEARCHES.

    Regards,
    Sven

  5. #5

    Default

    You are right, Swen :-) this is a data issue.

    The table size is extented because of the new columns added to the rows. I found this is the issue with the delimiter we used and when it was changed all worked fine.

    Thanks again!

    Your help is very immensely appreciated...

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.