Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: How to monitor memory?

  1. #1
    Join Date
    Sep 2014
    Posts
    175

    Default How to monitor memory?

    Hi there--

    I'm running into Java memory heap space issues in PDI ...

    It's odd, because on Computer A, there is no issue (2 GB allocated to PDI). But on computer B, there is an issue (also 2 GB allocated to PDI). Computer B has less memory overall though, so that's a potential factor.


    It's a job that has a "loop" with probably a small memory leak. However, it used to take 100 or 200 loops to cause this error --- now it takes 4 loops ... four!

    I'm guessing - since it's a shared computer -- external processes are coming in and sucking up memory during the job.

    Is there a tool that can monitor memory usage in PDI and other processes when it occurs? (at midnight). Aka not a live monitor necessarily, but something records memory usage of different components - not sure.


    By the way --- I also added a "dummy transformation" in the middle of the loop in my job. It does nothing but has the 'Advanced' settings checked: Clear list of result rows, clear the list of result files before execution. I thought this would be enough to plug the memory leak, but perhaps not.

    Anyone have any advice here?

    I'm not sure exactly how to scale back the memory usage of PDI here, other than making the job a "repeat every 2 seconds" instead of a loop, and killing PDI after the success condition, though that's definitely an extreme work-around.

  2. #2
    Join Date
    May 2016
    Posts
    282

    Default

    It's going to depend on your OS, for example, in linux I would program a cron task running free and/or top commands and writing the output to a file, so you can check the output near the time you run out of memory.
    Regards

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.