Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Out of Memory

  1. #1
    Join Date
    Nov 2007
    Posts
    10

    Default Out of Memory

    Hi,

    I am having some trouble to use an excel input after I upgraded to 3.0.1 build 534.

    For example, my input excel file contains 8000 lines, after I specified the filename and I try to obtain a sheetname, I got a out of memory error.

    I have 1GB of memory and it wasn't close to max out during operation, restarting kettle does not work.

    Please help.

    - Ken

    ------------
    Unable to open dialog for this step
    java.lang.OutOfMemoryError: Java heap space
    at jxl.read.biff.SheetReader.read(SheetReader.java:494)
    at jxl.read.biff.SheetImpl.readSheet(SheetImpl.java:686)
    at jxl.read.biff.WorkbookParser.getSheet(WorkbookParser.java:252)
    at org.pentaho.di.ui.trans.steps.excelinput.ExcelInputDialog.getSheets(ExcelInputDialog.java:1643)
    at org.pentaho.di.ui.trans.steps.excelinput.ExcelInputDialog$15.widgetSelected(ExcelInputDialog.java:1023)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.pentaho.di.ui.trans.steps.excelinput.ExcelInputDialog.open(ExcelInputDialog.java:1049)
    at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:124)
    at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:6383)
    at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:2153)
    at org.pentaho.di.ui.spoon.trans.TransGraph.access$100(TransGraph.java:116)
    at org.pentaho.di.ui.spoon.trans.TransGraph$4.mouseDoubleClick(TransGraph.java:312)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:841)
    at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:5589)
    at org.pentaho.di.ui.spoon.Spoon.run(Spoon.java:5685)
    at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:371)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.pentaho.commons.launcher.Launcher.main(Launcher.java:116)

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    SOoooo ... From spoon.bat e.g.

    Code:
    REM ******************************************************************
    REM ** Set java runtime options                                    
    REM ** Change 256m to higher values in case you run out of memory. 
    REM ******************************************************************
    Regards,
    Sven

  3. #3

    Default

    Tried that. Simple transform, selecting a bunch of rows (30), copying results to job and sending a copy to a CSV file. Set -mx512m. Still getting

    2007/12/28 15:23:06 - Write list to be processed to file.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : Unexpected error :
    2007/12/28 15:23:06 - Write list to be processed to file.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : java.lang.OutOfMemoryError: Java heap space
    2007/12/28 15:23:06 - Write list to be processed to file.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.convertStringToBinaryString(TextFileOutput.java:270)
    2007/12/28 15:23:06 - Write list to be processed to file.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.formatField(TextFileOutput.java:211)
    2007/12/28 15:23:06 - Write list to be processed to file.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.writeField(TextFileOutput.java:318)
    2007/12/28 15:23:06 - Write list to be processed to file.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.writeRowToFile(TextFileOutput.java:190)
    2007/12/28 15:23:06 - Write list to be processed to file.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.processRow(TextFileOutput.java:148)
    2007/12/28 15:23:06 - Write list to be processed to file.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.run(TextFileOutput.java:717)



    I shouldn't have to increase heap space so much just to launch one simple test transformation in preview mode, there is something wrong here.

  4. #4

    Default

    An updatre: when creating my Text File Output step, I used the "get fields" button to auto-populate the output fields. I had not noticed that the "Length" column for one of the fields was auto-set to an absurdly large number. That would do it. Once I corrected that things were fine.

    If it helps, the DB driver used is for PostgreSQL 8.1.

  5. #5
    Join Date
    May 2006
    Posts
    4,882

    Default

    That would solve your problem, but there's still another problem... the related JIRA contains an easily replayable out-of-memory.

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.