Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Running into OutOfMemory Issue

  1. #1

    Default Running into OutOfMemory Issue

    Hi,

    In my transformation i m trying to load 75,000 records into my database. My transformation crashes with OutOfMemory issue after loading 26,000 records.

    2008-08-18 14:07:57,673 INFO [STDOUT] ERROR 18-08 14:07:57,658 (LogWriter.java.Println:403) -Get RollUp Data.0 - org.pentaho.di.core.exception.KettleDatabaseException:
    An error occurred executing SQL:
    Select PRODUCT_TBL_ID from ROLLUP_PROCESSING where PRODUCT_TBL_ID = 2 AND NODE_TBL_ID = 1 AND DATE = '2055/08/10 00:00:00.000'
    Error; - nested throwable: (java.lang.OutOfMemoryError: Java heap space)
    at org.pentaho.di.core.database.Database.openQuery(Database.java:1752)
    at org.pentaho.di.core.database.Database.openQuery(Database.java:1676)
    at org.pentaho.di.core.database.Database.openQuery(Database.java:1671)
    at org.pentaho.di.core.database.Database.openQuery(Database.java:1658)
    at org.pentaho.di.core.database.Database.execStatements(Database.java:1600)
    at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:171)
    at org.pentaho.di.trans.steps.sql.ExecSQL.run(ExecSQL.java:277)
    Caused by: org.jboss.util.NestedSQLException: Error; - nested throwable: (java.lang.OutOfMemoryError: Java heap space)
    at org.jboss.resource.adapter.jdbc.WrappedConnection.checkException(WrappedConnection.java:557)
    at org.jboss.resource.adapter.jdbc.WrappedConnection.createStatement(WrappedConnection.java:149)
    at org.pentaho.di.core.database.Database.openQuery(Database.java:1718)
    ... 6 more


    Caused by: java.lang.OutOfMemoryError: Java heap space
    2008-08-18 14:08:05,724 INFO [STDOUT] ERROR 18-08 14:07:57,673 (LogWriter.java.Println:403) -Get Node Table ID.0 - org.pentaho.di.core.exception.KettleDatabaseException:
    An error occurred executing SQL:
    Select NODE_TBL_ID from NODE where LOCATION_ID = '129' AND ORG_ID = 2
    Error; - nested throwable: (java.lang.OutOfMemoryError: Java heap space)
    at org.pentaho.di.core.database.Database.openQuery(Database.java:1752)
    at org.pentaho.di.core.database.Database.openQuery(Database.java:1676)
    at org.pentaho.di.core.database.Database.openQuery(Database.java:1671)
    at org.pentaho.di.core.database.Database.openQuery(Database.java:1658)
    at org.pentaho.di.core.database.Database.execStatements(Database.java:1600)
    at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:171)
    at org.pentaho.di.trans.steps.sql.ExecSQL.run(ExecSQL.java:277)
    Caused by: org.jboss.util.NestedSQLException: Error; - nested throwable: (java.lang.OutOfMemoryError: Java heap space)
    at org.jboss.resource.adapter.jdbc.WrappedConnection.checkException(WrappedConnection.java:557)
    at org.jboss.resource.adapter.jdbc.WrappedConnection.createStatement(WrappedConnection.java:149)
    at org.pentaho.di.core.database.Database.openQuery(Database.java:1718)
    ... 6 more
    Caused by: java.lang.OutOfMemoryError: Java heap space
    2008-08-18 14:08:06,116 INFO [STDOUT] INFO 18-08 14:08:05,615 (LogWriter.java.Println:406) -Get RollUp Data.0 - Finished reading query, closing connection.


    Does kettle use application server's jvm memory or does it have its own internal heap memory for storing database records?

    Any pointers would help....

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    Which database type? Mysql?

    Regards,
    Sven

  3. #3

    Default Running into OutOfMemory Issue

    Ms -SQL Server...

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    Don't know enough about JBoss or SQL-server. I assume when you deploy whatever it is you deploy to run transformations, that you can also supply the maximum memory to be used.

    Regards,
    Sven

  5. #5

    Default

    Do you know where to increase size of memory for kettle?

    I found that in Kettle-Standalone, the Kettle.l4j.ini file have memory configuration settings.

    I have integrated Kettle in my application, so have just imported required kettle jars into my lib directory.

  6. #6
    Join Date
    May 2006
    Posts
    4,882

    Default

    The .ini is just if you use kettle.exe to start. Else you have to change the script spoon.sh/spoon.bat/pan.sh/... to change the heap size for the the JVM, but that only works in standalone mode of course.

    I expect something similar in an app server deployment, but as before no experience with JBoss... so we'll have to wait until someone comes along with more experience.

    Regards,
    Sven

  7. #7
    DEinspanjer Guest

    Default

    If you are driving Kettle inside your application by including the jars, then the memory configuration is completely dependent on your application's startup parameters. Kettle only uses the JVM memory naively. It doesn't do anything fancy.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.