Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: "Out of memory" on dummy job with Repeat enabled

  1. #1
    Join Date
    Sep 2008
    Posts
    26

    Default "Out of memory" on dummy job with Repeat enabled

    Hi,

    I have a simple dummy Job with a simple dummy transform that does nothing. All I do is repeat the transform on the Job Start. Spoon and Kitchen eventually run out of memory on the heap. This makes me believe that kettle does not clean up the transform memory each time a transform is called from the Job if the job is repeating.

    Is there a way to release the created transforms in a job when repeating, or to at least reuse the transform instance at a minimum? I created this simple job to prove the point. I have a lot more complex Job that repeats with about 5 transforms in it and it also crashes eventually.

    Alternatively, is there a way to loop through a process at a job level, calling the same transform without recreating the transform

    Thanks
    Des


    DummyTest.kjb
    DummyXform.ktr

  2. #2
    Join Date
    Sep 2008
    Posts
    26

    Default

    Hi,

    Was wondering if anybody could have a look at this please? This is currently a big problem and is influencing our decision to use Pentaho or not. If PDI cannot maintain memory leaks, then this is going to be a big problem.

    Is the issue with the architecture or simply the "repeating" option on a Job start? Is there another way to repeat the transforms in a job without constantly creating new instances of them?

    Thanks
    Des

  3. #3
    Join Date
    Sep 2009
    Posts
    810

    Default

    Hey Des,

    there were some issues related to memory leaks, but they were addressed afaik. Please check with the latest 4.1.3 snapshot here: http://ci.pentaho.com/view/Data%20In...1-SNAPSHOT.zip

    Cheers

    Slawo

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    The "Memory leak" in this case is caused by the logging back-end.
    I actually reproduced in in 4.1.3 and 4.2.0-M1.
    The solution for 4.2.0 is to run kitchen with the maxloglines option set.

    For some weird reason the KETTLE_MAX_LOG_SIZE_IN_LINES property is not picked up as I expected.

  5. #5
    Join Date
    Sep 2008
    Posts
    26

    Default

    Matt, Slawo

    Thanks for the responses. I will try kitchen with the log lines option.

    It there an bug report or ETA on the KETTLE_MAX_LOG_SIZE_IN_LINES issue?


    Thanks
    Des

  6. #6
    Join Date
    Sep 2008
    Posts
    26

    Default

    Matt,

    That seemed to do the trick, even on 4.1.

    ./kitchen.sh -version
    INFO 23-03 23:05:25,968 - Using "/var/folders/M4/M4QLZ99AHsuKdGVnliVao++++TI/-Tmp-/vfs_cache" as temporary files store.
    INFO 23-03 23:05:27,446 - Kitchen - Kettle version 4.1.0-stable, build 14410, build date : 2010-11-16 16.43.28


    The command I used was

    ./kitchen.sh -file /Users/des/MyJob.kjb -option maxloglines=1

    Never ran out of mem

    Please let me know when this could be fixed

    Thanks
    Des

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.