Hitachi Vantara Pentaho Community Forums
Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: java.lang.OutOfMemoryError: Java heap space ... How to solve?

  1. #1
    Join Date
    Aug 2008
    Posts
    563

    Default java.lang.OutOfMemoryError: Java heap space ... How to solve?

    Hi,
    I've tried to find some info about this on the forum, but wasn't successful with this.
    After I finally got my job running properly with the help of Sven and Matt I came accross another problem when running the job on huge tables:

    java.lang.OutOfMemoryError: Java heap space

    2009/03/30 15:38:00 - Table output.0 - Prepared statement : INSERT INTO rd_product_sales (customer_id, product_id, activation_date_id, activation_hour, activation_minute, cancelation_date_id, cancelation_hour, cancelation_minute, days_active, active, brand_id, country_id, operator_id, acquisition_media_id) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    2009/03/30 15:50:17 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : UnexpectedError: java.lang.OutOfMemoryError: Java heap space
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : java.lang.OutOfMemoryError: Java heap space
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at java.util.Arrays.copyOf(Unknown Source)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at java.lang.String.<init>(Unknown Source)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at com.mysql.jdbc.SingleByteCharsetConverter.toString(SingleByteCharsetConverter.java:286)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at com.mysql.jdbc.SingleByteCharsetConverter.toString(SingleByteCharsetConverter.java:262)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at com.mysql.jdbc.ResultSet.getStringInternal(ResultSet.java:4886)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at com.mysql.jdbc.ResultSet.getString(ResultSet.java:4810)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at org.pentaho.di.core.database.Database.getRow(Database.java:2817)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at org.pentaho.di.core.database.Database.getRow(Database.java:2784)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:137)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at org.pentaho.di.trans.step.BaseStep.runStepThread(BaseStep.java:2853)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at org.pentaho.di.trans.steps.tableinput.TableInput.run(TableInput.java:345)
    2009/03/30 15:50:58 - Calculator.0 - Finished processing (I=0, O=0, R=84828, W=84828, U=0, E=0)


    Can somebody tell me how I solve this?

    Thanks a lot,
    Diddy

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    That's easy to solve: give your transformation or job more memory to work with OR reduce your memory usage!

  3. #3
    Join Date
    Aug 2008
    Posts
    563

    Default

    Thanks a lot Matt for your answer!
    Can I do this somewhere within Kettle (if yes, how exactly) or would this be something that I have to do on the OS level?
    Best regards,
    Diddy

  4. #4

    Default

    Quote Originally Posted by diddy View Post
    Thanks a lot Matt for your answer!
    Can I do this somewhere within Kettle (if yes, how exactly) or would this be something that I have to do on the OS level?
    Best regards,
    Diddy
    If you are using Windows change the following line into the spoon.bat :
    Code:
    OPT="-Xmx256m -cp $CLASSPATH -Djava.library.path=$LIBPATH -DKETTLE_HOME=$KETTLE_HOME -DKETTLE_REPOSITORY=$KETTLE_REPOSITORY -DKETTLE_USER=$KETTLE_USER -DKETTLE_PASSWORD=$KETTLE_PASSWORD -DKETTLE_PLUGIN_PACKAGES=$KETTLE_PLUGIN_PACKAGES"
    If you are using Windows change the following line into the spoon.sh :
    Code:
    set OPT=-Xmx256m -cp %CLASSPATH% -Djava.library.path=libswt\win32\ -DKETTLE_HOME="%KETTLE_HOME%" -DKETTLE_REPOSITORY="%KETTLE_REPOSITORY%" -DKETTLE_USER="%KETTLE_USER%" -DKETTLE_PASSWORD="%KETTLE_PASSWORD%" -DKETTLE_PLUGIN_PACKAGES="%KETTLE_PLUGIN_PACKAGES%"
    Change the -Xmx256m option to -Xmx1024m for example, then launch kettle with this script.
    Louis Champion
    J2EE 5 Developper
    France, Paris

  5. #5
    Join Date
    Aug 2008
    Posts
    563

    Default

    Excellent! Thanks a lot for your help! I will try this out!
    Best regards,
    Diddy

  6. #6
    Join Date
    Aug 2008
    Posts
    563

    Default

    I have tried to change this but still I get the same error message. Is there anything else that I can do?
    I am loading a table which has about 400,000 rows in one go (I had to disable streaming). Do I need more than 1024 MB RAM for this?

  7. #7
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Obviously, since you still run out of memory.

  8. #8
    DEinspanjer Guest

    Default

    Quote Originally Posted by diddy View Post
    I have tried to change this but still I get the same error message. Is there anything else that I can do?
    I am loading a table which has about 400,000 rows in one go (I had to disable streaming). Do I need more than 1024 MB RAM for this?
    You appear to have about 15 integer or big integer fields. With 400000 rows, that translates to just under one GB of memory just for the raw data itself. You also have to take into account JDBC memory overhead and lots of other things. 1 GB of allocated data is likely way to small for this task.

    Normally, streaming result sets is the answer to this problem. It allows your JDBC client (Kettle in this case) to pull batches of rows into memory one at a time. Any particular reason you had to disable it?

  9. #9
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    MySQL only allows you to open a single such streaming query. If you want to open multiple (unique connections) it fails.

  10. #10
    DEinspanjer Guest

    Default

    Oh yeah. I've run into that limitation before.

    Any chance you could write this data out to a file instead and then use the bulk load job entry?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.