Hitachi Vantara Pentaho Community Forums
Results 1 to 16 of 16

Thread: java.lang.OutOfMemoryError: Java heap space ... How to solve?

  1. #1
    Join Date
    Aug 2008
    Posts
    563

    Default java.lang.OutOfMemoryError: Java heap space ... How to solve?

    Hi,
    I've tried to find some info about this on the forum, but wasn't successful with this.
    After I finally got my job running properly with the help of Sven and Matt I came accross another problem when running the job on huge tables:

    java.lang.OutOfMemoryError: Java heap space

    2009/03/30 15:38:00 - Table output.0 - Prepared statement : INSERT INTO rd_product_sales (customer_id, product_id, activation_date_id, activation_hour, activation_minute, cancelation_date_id, cancelation_hour, cancelation_minute, days_active, active, brand_id, country_id, operator_id, acquisition_media_id) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    2009/03/30 15:50:17 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : UnexpectedError: java.lang.OutOfMemoryError: Java heap space
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : java.lang.OutOfMemoryError: Java heap space
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at java.util.Arrays.copyOf(Unknown Source)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at java.lang.String.<init>(Unknown Source)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at com.mysql.jdbc.SingleByteCharsetConverter.toString(SingleByteCharsetConverter.java:286)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at com.mysql.jdbc.SingleByteCharsetConverter.toString(SingleByteCharsetConverter.java:262)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at com.mysql.jdbc.ResultSet.getStringInternal(ResultSet.java:4886)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at com.mysql.jdbc.ResultSet.getString(ResultSet.java:4810)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at org.pentaho.di.core.database.Database.getRow(Database.java:2817)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at org.pentaho.di.core.database.Database.getRow(Database.java:2784)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:137)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at org.pentaho.di.trans.step.BaseStep.runStepThread(BaseStep.java:2853)
    2009/03/30 15:50:33 - Product Cancelations.0 - ERROR (version 3.2.0-M2, build 10376 from 2009-03-16 04.27.19 by tomcat) : at org.pentaho.di.trans.steps.tableinput.TableInput.run(TableInput.java:345)
    2009/03/30 15:50:58 - Calculator.0 - Finished processing (I=0, O=0, R=84828, W=84828, U=0, E=0)


    Can somebody tell me how I solve this?

    Thanks a lot,
    Diddy

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    That's easy to solve: give your transformation or job more memory to work with OR reduce your memory usage!

  3. #3
    Join Date
    Aug 2008
    Posts
    563

    Default

    Thanks a lot Matt for your answer!
    Can I do this somewhere within Kettle (if yes, how exactly) or would this be something that I have to do on the OS level?
    Best regards,
    Diddy

  4. #4

    Default

    Quote Originally Posted by diddy View Post
    Thanks a lot Matt for your answer!
    Can I do this somewhere within Kettle (if yes, how exactly) or would this be something that I have to do on the OS level?
    Best regards,
    Diddy
    If you are using Windows change the following line into the spoon.bat :
    Code:
    OPT="-Xmx256m -cp $CLASSPATH -Djava.library.path=$LIBPATH -DKETTLE_HOME=$KETTLE_HOME -DKETTLE_REPOSITORY=$KETTLE_REPOSITORY -DKETTLE_USER=$KETTLE_USER -DKETTLE_PASSWORD=$KETTLE_PASSWORD -DKETTLE_PLUGIN_PACKAGES=$KETTLE_PLUGIN_PACKAGES"
    If you are using Windows change the following line into the spoon.sh :
    Code:
    set OPT=-Xmx256m -cp %CLASSPATH% -Djava.library.path=libswt\win32\ -DKETTLE_HOME="%KETTLE_HOME%" -DKETTLE_REPOSITORY="%KETTLE_REPOSITORY%" -DKETTLE_USER="%KETTLE_USER%" -DKETTLE_PASSWORD="%KETTLE_PASSWORD%" -DKETTLE_PLUGIN_PACKAGES="%KETTLE_PLUGIN_PACKAGES%"
    Change the -Xmx256m option to -Xmx1024m for example, then launch kettle with this script.
    Louis Champion
    J2EE 5 Developper
    France, Paris

  5. #5
    Join Date
    Aug 2008
    Posts
    563

    Default

    Excellent! Thanks a lot for your help! I will try this out!
    Best regards,
    Diddy

  6. #6
    Join Date
    Aug 2008
    Posts
    563

    Default

    I have tried to change this but still I get the same error message. Is there anything else that I can do?
    I am loading a table which has about 400,000 rows in one go (I had to disable streaming). Do I need more than 1024 MB RAM for this?

  7. #7
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Obviously, since you still run out of memory.

  8. #8
    DEinspanjer Guest

    Default

    Quote Originally Posted by diddy View Post
    I have tried to change this but still I get the same error message. Is there anything else that I can do?
    I am loading a table which has about 400,000 rows in one go (I had to disable streaming). Do I need more than 1024 MB RAM for this?
    You appear to have about 15 integer or big integer fields. With 400000 rows, that translates to just under one GB of memory just for the raw data itself. You also have to take into account JDBC memory overhead and lots of other things. 1 GB of allocated data is likely way to small for this task.

    Normally, streaming result sets is the answer to this problem. It allows your JDBC client (Kettle in this case) to pull batches of rows into memory one at a time. Any particular reason you had to disable it?

  9. #9
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    MySQL only allows you to open a single such streaming query. If you want to open multiple (unique connections) it fails.

  10. #10
    DEinspanjer Guest

    Default

    Oh yeah. I've run into that limitation before.

    Any chance you could write this data out to a file instead and then use the bulk load job entry?

  11. #11
    Join Date
    Aug 2008
    Posts
    563

    Default

    Quote Originally Posted by MattCasters View Post
    MySQL only allows you to open a single such streaming query. If you want to open multiple (unique connections) it fails.
    ... this is a good piece of information!

    I will try to first import the data with one job into a text file.
    The second job will do the transformations than and export it to the final database. For this step I can use streaming then. Let's see if this helps.

  12. #12
    Join Date
    Aug 2008
    Posts
    563

    Default

    So, now finally all the problems are solved!
    I split my original transaction into two seperate transaction ...

    • the first one reading the data in from MySQL and exporting it to a text file
    • the second one reading the text file in, doing some transformation and exporting the data to another MySQL server

    The main job uses the first transformation to import several tables.

    Well, it seems like it is not possible to do this all in one transformation with MySQL.
    Now, I also could turn 'streaming' back on.
    Thanks a lot all for you help!
    Best regards,
    Diddy

  13. #13

    Default log level

    Try changing the log level.
    We have changed level debug to minimal and it works.
    We think the log level is the cause of the error in our job.

  14. #14
    Join Date
    Mar 2015
    Posts
    18

    Default

    I had a similar problem - I was pulling data from an XML datatype field (Postgres) and converting the elements into a lot of fields which I was casting to text. I wasn't setting a length for the text however, so by default the database would assign the largest possible value (2147483647). This, along with a bunch of BigNumber data types was causing Pentaho to run out of memory during the transformation run. I ended up casting during the select statement and included the lengths, so it became:

    select
    column_a :: varchar(4) as column_a,
    column_b :: varchar(8) as column_b,
    ...
    from table
    ...

    This solved my problem

    Ryan.

  15. #15
    Join Date
    Aug 2015
    Posts
    1

    Default java error - CSV Output to S3 Bucket

    I was experiencing the same issue as rakesh411 and did want to point out that as opposed to casting your select input(s) on some transforms there is an option to select the minimum width of the values in your returned data set. In my use case.....I was outputting the data to an S3 bucket. Prior to minimizing the width, I was getting the same lovely java error.
    Name:  2015-09-15_17-18-26.jpg
Views: 822
Size:  31.5 KB


    regards
    Attached Images Attached Images  

  16. #16
    Join Date
    Jan 2016
    Posts
    10

    Default

    Hello everybody! I'm trying to read a mbox file with Pentaho DI 6, and the log show me this:
    2016/09/12 08:48:44 - cfgbuilder - Warning: The configuration parameter [org] is not supported by the default configuration builder for scheme: sftp2016/09/12 08:50:59 - C:\Users\fangonzalez\Desktop\Pentaho\trans.ktr : trans - Dispatching started for transformation [C:\Users\fangonzalez\Desktop\Pentaho\trans.ktr : trans]
    2016/09/12 08:57:06 - Email messages input.0 - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : UnexpectedError:
    2016/09/12 08:57:06 - Email messages input.0 - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : java.lang.OutOfMemoryError: Java heap space
    2016/09/12 08:57:06 - Email messages input.0 - at java.nio.HeapByteBuffer.<init>(Unknown Source)
    2016/09/12 08:57:06 - Email messages input.0 - at java.nio.ByteBuffer.allocate(Unknown Source)
    2016/09/12 08:57:06 - Email messages input.0 - at net.fortuna.mstor.data.MboxFile.read(MboxFile.java:280)
    2016/09/12 08:57:06 - Email messages input.0 - at net.fortuna.mstor.data.MboxFile.getMessageAsStream(MboxFile.java:428)
    2016/09/12 08:57:06 - Email messages input.0 - at net.fortuna.mstor.connector.mbox.MboxFolder.getMessageAsStream(MboxFolder.java:261)
    2016/09/12 08:57:06 - Email messages input.0 - at net.fortuna.mstor.MStorFolder.getMessage(MStorFolder.java:407)
    2016/09/12 08:57:06 - Email messages input.0 - at javax.mail.Folder.getMessages(Folder.java:947)
    2016/09/12 08:57:06 - Email messages input.0 - at org.pentaho.di.job.entries.getpop.MailConnection.retrieveMessages(MailConnection.java:674)
    2016/09/12 08:57:06 - Email messages input.0 - at org.pentaho.di.trans.steps.mailinput.MailInput.openNextFolder(MailInput.java:357)
    2016/09/12 08:57:06 - Email messages input.0 - at org.pentaho.di.trans.steps.mailinput.MailInput.getOneRow(MailInput.java:214)
    2016/09/12 08:57:06 - Email messages input.0 - at org.pentaho.di.trans.steps.mailinput.MailInput.processRow(MailInput.java:75)
    2016/09/12 08:57:06 - Email messages input.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
    2016/09/12 08:57:06 - Email messages input.0 - at java.lang.Thread.run(Unknown Source)
    2016/09/12 08:57:06 - Email messages input.0 - Finished processing (I=0, O=0, R=0, W=0, U=0, E=1)
    2016/09/12 08:57:06 - C:\Users\fangonzalez\Desktop\Pentaho\trans.ktr : trans - Transformation detected one or more steps with errors.
    2016/09/12 08:57:06 - C:\Users\fangonzalez\Desktop\Pentaho\trans.ktr : trans - Transformation is killing the other steps!
    I try to do what Campi says seven years ago, change the value in the spoon.bat and spoon.sh files, to extend the memory capacity. Buuut, instead of the lines that he show, I find this:
    if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xms1024m" "-Xmx2048m" "-XX:MaxPermSize=256m"
    I need to change the -XX:MaxPermSize var, correct?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.