Hitachi Vantara Pentaho Community Forums
Results 1 to 11 of 11

Thread: Error java.lang.OutOfMemoryError: Java heap space ejecutando una transformación

  1. #1
    Join Date
    Mar 2010
    Posts
    28

    Default Error java.lang.OutOfMemoryError: Java heap space ejecutando una transformación

    Buenos días.

    He creado una transformación para cargar un excel en una tabla. Y me da el siguiente error.

    No entiendo porque dá un error de espacio. Es muy grande el excel?.


    Adjunto el error.

    2010/11/30 12:22:13 - CargaDadesVendes - Iniciado despacho de la transformación [CargaDadesVendes]
    2010/11/30 12:22:13 - CargaDadesVendes - Esta transformación se puede repetir con fecha: 2010/11/30 12:22:13
    2010/11/30 12:22:14 - Salida Tabla.0 - Connected to database [ES-D-GRP81J] (commit=100)
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : UnexpectedError: java.lang.OutOfMemoryError: Java heap space
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : java.lang.OutOfMemoryError: Java heap space
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : at jxl.read.biff.MulRKRecord.<init>(MulRKRecord.java:76)
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : at jxl.read.biff.SheetReader.read(SheetReader.java:458)
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : at jxl.read.biff.SheetImpl.readSheet(SheetImpl.java:686)
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : at jxl.read.biff.WorkbookParser.getSheet(WorkbookParser.java:252)
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : at jxl.read.biff.WorkbookParser.getSheet(WorkbookParser.java:285)
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : at org.pentaho.di.trans.steps.excelinput.ExcelInput.getRowFromWorkbooks(ExcelInput.java:533)
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : at org.pentaho.di.trans.steps.excelinput.ExcelInput.processRow(ExcelInput.java:405)
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : at org.pentaho.di.trans.step.BaseStep.runStepThread(BaseStep.java:2664)
    2010/11/30 12:22:41 - Entrada Excel.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 12:32:36) : at org.pentaho.di.trans.steps.excelinput.ExcelInput.run(ExcelInput.java:803)
    2010/11/30 12:22:41 - Entrada Excel.0 - Procesamiento finalizado (I=0, O=0, R=0, W=0, U=0, E=1
    2010/11/30 12:22:41 - CargaDadesVendes - Transformación detectada
    2010/11/30 12:22:41 - CargaDadesVendes - Transformación está matando los otros pasos!
    2010/11/30 12:22:41 - Spoon - La transformación ha finalizado!!


    Esta transformación ya me había funcionado anteriormente.

    2010/11/12 08:00:51 - CargaDadesVendes - Loading transformation from XML file [file:///C:/Cargas/CargasAutomatizadasdesdeExcel/cargaDadesVendes.ktr]
    2010/11/12 08:00:51 - CargaDadesVendes - Iniciado despacho de la transformación [CargaDadesVendes]
    2010/11/12 08:00:51 - CargaDadesVendes - Esta transformación se puede repetir con fecha: 2010/11/12 08:00:51
    2010/11/12 08:00:51 - Salida Tabla.0 - Connected to database [ES-D-GRP81J] (commit=100)
    2010/11/12 08:01:46 - Entrada Excel.0 - Procesamiento finalizado (I=0, O=0, R=0, W=50000, U=50000, E=0
    2010/11/12 08:01:57 - Salida Tabla.0 - linenr 50000
    2010/11/12 08:01:57 - Salida Tabla.0 - Procesamiento finalizado (I=0, O=50000, R=50000, W=50000, U=50000, E=0


    Saludos.
    Last edited by cgonzalezc; 11-30-2010 at 07:28 AM.

  2. #2
    Join Date
    Feb 2009
    Posts
    321

    Default

    proba modificando la memoria dentro del spoon.exe, fijate que por default esta en 256, el error Java heap space es cuando excedes la memoria que tenes predeterminada...

  3. #3
    Join Date
    Mar 2010
    Posts
    28

    Default

    ¿Donde se modifica el tema de la memoria?, lo estoy buscando pero no lo encuentro.


    Gracias

  4. #4
    Join Date
    Feb 2009
    Posts
    321

    Default

    este es el kitchen.bat, donde dice el 512 en rojo resaltado.. cambiale a 1024 (1 giga)



    REM **************************************************
    REM ** Set console window properties **
    REM **************************************************
    REM TITLE Chef console
    REM COLOR F0
    REM **************************************************
    REM ** Make sure we use the correct J2SE version! **
    REM ** Uncomment the PATH line in case of trouble **
    REM **************************************************
    REM set PATH=C:\j2sdk1.4.2_01\bin;.;%PATH%
    REM **************************************************
    REM ** Libraries used by Kettle: **
    REM **************************************************
    set CLASSPATH=.
    REM ******************
    REM KETTLE Library
    REM ******************
    set CLASSPATH=%CLASSPATH%;lib\kettle-core.jar
    set CLASSPATH=%CLASSPATH%;lib\kettle-db.jar
    set CLASSPATH=%CLASSPATH%;lib\kettle-engine.jar
    REM **********************
    REM External Libraries
    REM **********************
    REM Loop the libext directory and add the classpath.
    REM The following command would only add the last jar: FOR %%F IN (libext\*.jar) DO call set CLASSPATH=%CLASSPATH%;%%F
    REM So the circumvention with a subroutine solves this ;-)
    FOR %%F IN (libext\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\JDBC\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\webservices\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\commons\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\web\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\pentaho\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\spring\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\mondrian\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\salesforce\*.jar) DO call :addcp %%F
    goto extlibe
    :addcp
    set CLASSPATH=%CLASSPATH%;%1
    goto :eof
    :extlibe

    REM *****************
    REM SWT Libraries
    REM *****************
    set CLASSPATH=%CLASSPATH%;libswt\runtime.jar
    set CLASSPATH=%CLASSPATH%;libswt\jface.jar
    set CLASSPATH=%CLASSPATH%;libswt\win32\swt.jar
    REM **********************
    REM Collect arguments
    REM **********************
    set _cmdline=
    :TopArg
    if %1!==! goto EndArg
    set _cmdline=%_cmdline% %1
    shift
    goto TopArg
    :EndArg
    REM ******************************************************************
    REM ** Set java runtime options **
    REM ** Change 512m to higher values in case you run out of memory. **
    REM ******************************************************************
    set OPT=-Xmx512m -cp %CLASSPATH% -Djava.library.path=libswt\win32\ -DKETTLE_HOME="%KETTLE_HOME%" -DKETTLE_REPOSITORY="%KETTLE_REPOSITORY%" -DKETTLE_USER="%KETTLE_USER%" -DKETTLE_PASSWORD="%KETTLE_PASSWORD%" -DKETTLE_PLUGIN_PACKAGES="%KETTLE_PLUGIN_PACKAGES%" -DKETTLE_LOG_SIZE_LIMIT="%KETTLE_LOG_SIZE_LIMIT%"
    REM ***************
    REM ** Run... **
    REM ***************
    java %OPT% org.pentaho.di.kitchen.Kitchen %_cmdline%

  5. #5
    Join Date
    Mar 2010
    Posts
    28

    Default

    Buenos días.

    Lo que pasa que no estoy usando la herramienta kitchen.

    Lo ejecuto directamente desde la interfaz gráfica del Spoon.


    Saludos.

  6. #6
    Join Date
    Feb 2009
    Posts
    321

    Default

    es lo mismo edita el spoon prueba primero con 512 y sino con 1024 ( este es el edit del spoon)

    @echo off
    REM **************************************************
    REM Check for Windows 2000, since the command line length is limited
    REM **************************************************
    ver | find "Windows 2000" >nul
    if errorlevel 1 goto nowin2k
    echo Attention: You are using Windows 2000. Please see
    echo http://wiki.pentaho.org/display/EAI/Windows+2000
    echo when Spoon does not start. After fixing the problem
    echo you can delete the pause line in your Spoon.bat file.
    pause
    :nowin2k
    REM **************************************************
    REM ** Set console window properties **
    REM **************************************************
    REM TITLE Spoon console
    REM COLOR F0
    REM **************************************************
    REM ** Make sure we use the correct J2SE version! **
    REM ** Uncomment the PATH line in case of trouble **
    REM **************************************************
    REM set PATH=C:\j2sdk1.4.2_01\bin;.;%PATH%
    REM **************************************************
    REM ** Libraries used by Kettle: **
    REM **************************************************
    set CLASSPATH=.
    REM ******************
    REM KETTLE Library
    REM ******************
    set CLASSPATH=%CLASSPATH%;lib\kettle-core.jar
    set CLASSPATH=%CLASSPATH%;lib\kettle-db.jar
    set CLASSPATH=%CLASSPATH%;lib\kettle-engine.jar
    set CLASSPATH=%CLASSPATH%;lib\kettle-ui-swt.jar
    REM **********************
    REM External Libraries
    REM **********************
    REM Loop the libext directory and add the classpath.
    REM The following command would only add the last jar: FOR %%F IN (libext\*.jar) DO call set CLASSPATH=%CLASSPATH%;%%F
    REM So the circumvention with a subroutine solves this ;-)
    FOR %%F IN (libext\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\JDBC\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\webservices\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\commons\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\web\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\pentaho\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\spring\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\jfree\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\mondrian\*.jar) DO call :addcp %%F
    FOR %%F IN (libext\salesforce\*.jar) DO call :addcp %%F

    goto extlibe
    :addcp
    set CLASSPATH=%CLASSPATH%;%1
    goto :eof
    :extlibe
    REM **********************
    REM Collect arguments
    REM **********************
    set _cmdline=
    :TopArg
    if %1!==! goto EndArg
    set _cmdline=%_cmdline% %1
    shift
    goto TopArg
    :EndArg

    REM *****************
    REM SWT Libraries
    REM *****************
    set CLASSPATH=%CLASSPATH%;libswt\runtime.jar
    set CLASSPATH=%CLASSPATH%;libswt\jface.jar
    set CLASSPATH=%CLASSPATH%;libswt\common.jar
    set CLASSPATH=%CLASSPATH%;libswt\commands.jar
    set CLASSPATH=%CLASSPATH%;libswt\win32\swt.jar
    REM ******************************************************************
    REM ** Set java runtime options **
    REM ** Change 256m to higher values in case you run out of memory. **
    REM ******************************************************************
    set OPT=-Xmx256m -cp %CLASSPATH% -Djava.library.path=libswt\win32\ -DKETTLE_HOME="%KETTLE_HOME%" -DKETTLE_REPOSITORY="%KETTLE_REPOSITORY%" -DKETTLE_USER="%KETTLE_USER%" -DKETTLE_PASSWORD="%KETTLE_PASSWORD%" -DKETTLE_PLUGIN_PACKAGES="%KETTLE_PLUGIN_PACKAGES%" -DKETTLE_LOG_SIZE_LIMIT="%KETTLE_LOG_SIZE_LIMIT%"
    REM ***************
    REM ** Run... **
    REM ***************
    start javaw %OPT% org.pentaho.di.ui.spoon.Spoon %_cmdline%

  7. #7
    Join Date
    Mar 2010
    Posts
    28

    Default

    Hola, me modificado tanto el Kitchen como el Spoon, pero sigue dandome el mismo error.


    Saludos.



    Carlos

  8. #8

    Default

    Estoy trabajando con el spoon, y una transformacion me daba error con java heap space.
    Incremente el mxm de 512 a 1024 y el error persiste.

    Cual es el limite para la memoria xmx, depende de la memoria RAM?

    Parte del log:
    011/01/03 08:45:14 - Tabla Entrada.0 - linenr 700000
    2011/01/03 08:45:14 - Selecciona.0 - linenr 700000
    2011/01/03 08:45:14 - d_servicio.0 - linenr 700000
    2011/01/03 08:45:14 - d_contrato.0 - linenr 700000
    2011/01/03 08:45:15 - d_regimen.0 - linenr 700000
    2011/01/03 08:45:16 - Calcula edad.0 - Linenr 700000
    2011/01/03 08:45:16 - ordena edad.0 - Linenr 700000
    2011/01/03 08:46:27 - Tabla Entrada.0 - linenr 750000
    2011/01/03 08:46:27 - Selecciona.0 - linenr 750000
    2011/01/03 08:46:27 - d_servicio.0 - linenr 750000
    2011/01/03 08:46:27 - d_contrato.0 - linenr 750000
    2011/01/03 08:46:27 - d_regimen.0 - linenr 750000
    2011/01/03 08:46:27 - Calcula edad.0 - Linenr 750000
    2011/01/03 08:46:27 - ordena edad.0 - Linenr 750000
    2011/01/03 08:54:21 - Tabla Entrada.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : UnexpectedError:
    2011/01/03 08:55:24 - Calcula edad.0 - Finished processing (I=0, O=0, R=784198, W=784197, U=0, E=0)
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : UnexpectedError:
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : java.lang.OutOfMemoryError: Java heap space
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : java.util.Arrays.copyOf(Unknown Source)
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : java.util.Arrays.copyOf(Unknown Source)
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : java.util.ArrayList.toArray(Unknown Source)
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : java.util.Collections.sort(Unknown Source)
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : org.pentaho.di.trans.steps.sort.SortRows.quickSort (SortRows.java:568)
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : org.pentaho.di.trans.steps.sort.SortRows.addBuffer (SortRows.java:90)
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : org.pentaho.di.trans.steps.sort.SortRows.processRo w(SortRows.java:408)
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : org.pentaho.di.trans.step.RunThread.run(RunThread. java:40)
    2011/01/03 08:55:35 - ordena edad.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : java.lang.Thread.run(Unknown Source)

    2011/01/03 08:55:35 - ordena edad.0 - Finished processing (I=0, O=0, R=784197, W=0, U=0, E=1)
    2011/01/03 08:55:35 - h_facturacion_servicios_dinamica_pdi2 - h_facturacion_servicios_dinamica_pdi2
    2011/01/03 08:55:36 - h_facturacion_servicios_dinamica_pdi2 - h_facturacion_servicios_dinamica_pdi2
    2011/01/03 08:55:36 - Regimen.0 - Finished processing (I=0, O=0, R=784202, W=784201, U=0, E=0)
    2011/01/03 08:55:36 - Tabla Entrada.0 - Finished reading query, closing connection.
    2011/01/03 08:55:36 - Selecciona.0 - Finished processing (I=0, O=0, R=784342, W=784342, U=0, E=0)
    2011/01/03 08:55:36 - Tabla Entrada.0 - Finished processing (I=784442, O=0, R=0, W=784441, U=0, E=1)
    2011/01/03 08:55:36 - d_servicio.0 - Finished processing (I=3578, O=383, R=784224, W=784223, U=0, E=0)
    2011/01/03 08:55:36 - d_regimen.0 - Finished processing (I=12, O=0, R=784200, W=784199, U=0, E=0)
    2011/01/03 08:55:36 - d_contrato.0 - Finished processing (I=231, O=1, R=784202, W=784202, U=0, E=0)
    2011/01/03 08:55:36 - h_facturacion_servicios_dinamica_pdi2 - h_facturacion_servicios_dinamica_pdi2
    2011/01/03 08:55:36 - h_facturacion_servicios_dinamica_pdi2 - h_facturacion_servicios_dinamica_pdi2

  9. #9
    Join Date
    Feb 2009
    Posts
    321

    Default

    probaron con hace un paso en varios? tirando threads q divide los recursos de la memoria ... click derecho sobre el step opcion "change number of copies to start" y jugar con el numero de copias en paralelo que crea. Probate eso y vemos luego
    saludos
    Hernan

  10. #10

    Default

    Hernan, pero como es eso de los threads? que numero hay que poner donde dice "number of copies (1 or higher)" y esto hay que ponerlo en todos los pasos (steps) o solo en el paso de entrada (table input)???

  11. #11
    Join Date
    Jul 2010
    Posts
    23

    Default

    Hernan, yo tengo el mismo problema, ya he probado pondiendo el número de threads en 2, 10, 100 y da igual. El problema es que al archivo excel .xlsx tiene 1.000.000 de filas y al parecer el etl trata de cargarlo a memoria y alli da problema de java heap. El mismo archivo pero con 1000 filas si me funciona.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.