Hitachi Vantara Pentaho Community Forums
Results 1 to 17 of 17

Thread: OutOfMemoryError: heap space

  1. #1
    Join Date
    Apr 2007
    Posts
    16

    Default OutOfMemoryError: heap space

    Hello,
    This may be a completely bonehead move on my part, but I've spent a lot of time on this today and I'm just not seeing what else I can do.
    Here's the problem. I'm running transformations and jobs from the API. I have it set up to run on a pre-determined time interval via the Quartz scheduler. The problem is, each time I execute a job, it seems to consume a little more heap space memory and not let it go. So I end up with a perpetually increasing memory consumption until the app eventually crashes with OutOfMemoryError: heap space.

    I initially suspected it was something in my transformations, so I removed all transformations until I had a job with just a start & dummy step. The rate of the upward memory trend decreased significantly but still appeared to have the overall problem. Then I changed it to just creating the job and not executing it... still appeared to have the problem.
    Now I don't even create the job. I'm just creating a JobMeta and doing nothing with it.

    For the purpose of testing , I made the execute frequency to once every 3 seconds. I'm just assuming that I'm still having the problem by looking at the memory trend line in JConsole. If the peaks continue to rise with each run, then I assume I will eventually crash with OutOfMemoryError again... ?

    Please correct my assumptions or offer any suggestions.

    Thanks!




    Here's the code broken down into its simplest form. I've tested this method with no code in it all just to eliminate any other potential causes and I see flat memory usage when I do that.


    Code:
    private static String parentThreadName;
    static {
      EnvUtil.environmentInit();
      parentThreadName = Thread.currentThread().getName();
      StepLoader.getInstance().read();
      JobEntryLoader.getInstance().read(); }
    
    
    public void executeJob(File file) throws Exception {
      Repository rep = null;
      LogWriter logWriter = null;
      JobMeta meta = new JobMeta(logWriter , file.getPath(), rep);
            meta.clear();
            meta = null; }
    Attached Images Attached Images  

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    There was a little memory leakage with the Variables in 2.5. Which is fixed in 3.0 but the change too big to backport to 2.5.

    Regards,
    Sven

  3. #3
    Join Date
    Apr 2007
    Posts
    16

    Default

    Thanks for the heads up. I thought that I could get around the problem by creating the JobMeta and Job objects only once and re-using them for each execution. But when I do that, I get the following error after a few runs...

    Could not read from "file:///C:/tomcat/${Internal.Job.Filename.Directory}/test.ktr" because it is a not a file.
    The file clearly does exist as it does execute fine otherwise. I'm guessing that this 'Internal.Job.Filename.Directory' variable is not initialized in subsequent runs.
    Any ideas?
    Thanks

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    The variables have been completely reviewed in v3.0 because of the problems you're seeing with them. Problem is that the v3 is not yet "stable" enough to be really used.

    Regards,
    Sven

  5. #5

    Default

    I have experienced the OutOfMemory, and have resolved it for my scenario here:

    http://forums.pentaho.org/showthread.php?t=54359

    However, I am not sure of the impact of the changes I posted, so take those for granted (I have not experienced any issues on my production, but that's only one usecase).

  6. #6
    Join Date
    Apr 2007
    Posts
    16

    Default

    Thanks!!
    Simply adding your endProcessing() line to my code seems to have done the trick.

    Result result = job.execute();
    job.waitUntilFinished();
    job.endProcessing("end", result);

  7. #7
    Join Date
    Apr 2007
    Posts
    16

    Default

    FYI: I spoke a little prematurely. Although it did make a big improvement (memory leak is -much- slower), I believe that I am witnessing a more gradual leak now.
    Looks like I'm going to have to figure out a way to decouple this from my app and run it from pan.

  8. #8

    Default

    yeah, the gradual leak I've seen is probably related to the variable handling Sven mentioned and the Kettle guys are correcting in 3.0.

    Could you share how much of an impact the gradual memory leak is? Although I recognize it, it doesn't seem to have been a showstopper in my tests, nor on my production system.

    I guess to be more precise - have you seen the memory leak occur, but the GC eventually clean it up anyway opposed to an OutOfMemory problem related to the leak?

  9. #9
    Join Date
    May 2006
    Posts
    4,882

    Default

    The memory leak (besides the kettlecomponent) won't go away by gc... they are values hanging around in a singleton hashmap, and the garbage collector is not aware these can be reaped.

    Regards,
    Sven

  10. #10
    Join Date
    Apr 2007
    Posts
    16

    Default

    dhartford,
    To answer you question about the impact the gradual leak has... Previously, with the rapid leak, I would quickly crash with out-of-mem in about half a day. I think now it would take much longer.. how much longer, I don't know. But as Sven, mentioned, GC wasn't helping (and I confirmed that by explicitly calling GC from JConsole), so it would be only a matter of time before it happens again.
    I guess it just all depends on how often you run your tansformations vs how often your app gets restarted. I'm now running my job from Kitchen at an interval using Windows scheduler. I haven't really optimized it for this.... an entire separate copy of my app essentially gets fired up with each execution, so it uses a lot of horsepower doing so. It's not pretty, but I think it'll hold me over until v3 is ready.

  11. #11

    Default

    I appear to be facing the same problem as reported here. Did you find a solution ?

    1. Do you get the StepLoader and JobEntryLoader only at startup?
    2. I di dnot notice the meta.clear or the job.endProcessing("end", result); methods. I will add them to my code. Are there other methods that can be used to release resources?
    3. What is the variable leask that Sven mentioned?
    Thanks

  12. #12
    Join Date
    May 2006
    Posts
    4,882

    Default

    For 3) in 2.5 variables were stored in global hashmap keyed by threadname, but since the hashmap is not tied to the jobs/transformation they don't always get cleared and leak a little bit of memory. If you know at a certain point that no jobs/transformations are running you could probably clear the KettleVariables manually. In v3 variables are tied to job entries and transformation steps, so that when they get reaped the variables go with them.
    But it's quite a big change so it couldn't be ported to 2.5

    Regards,
    Sven

  13. #13

    Default

    Thanks - how would I clear these manually.

    What are the variables that we are referencing? Are these the environment variables or something else? If environment variables I have to assume that the leak is minimal and limited.

  14. #14
    Join Date
    May 2006
    Posts
    4,882

    Default

    It's be.ibridge.kettle.core.KettleVariables, but no real way of clearing them without getting the properties object out of it and clearing that. As in the previous part of this thread this is not a huge leak... it just grows steadily: it contains all variables in your Java System environment + the kettle.properties ones.

    For now I would use the job.endProcessing() and leave the variables.

    Regards,
    Sven

  15. #15
    Join Date
    Apr 2007
    Posts
    16

    Default

    Hi, can you guys tell me if this issue is fixed in 3.0 yet?
    In other words, can I repeatedly run jobs & transformations from Java without with the eventual OutOfMemoryError?

    Thanks!

  16. #16
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    I really think that this issue is fixed, yes.
    See for yourself later today.

  17. #17
    Join Date
    Sep 2007
    Posts
    13

    Default I get this all the time in 3.0

    Is it really fixed or is a configuration issue on my end?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.