Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: How much memory consumption is normal? (PC)

  1. #1
    Join Date
    Dec 2016
    Posts
    27

    Default How much memory consumption is normal? (PC)

    Hi,

    I'm using PDI on my work issue Lenovo x240 with 8GB non-upgradable RAM

    I'm regularly consuming over 2GB memory which feel excessive. Am I doing something wrong? This impairs my ability to use both PDI and other applications because of resource constraints what with all of the security bloat on my laptop. Is there a control somewhere which would allow me to clear out the cache for a given transformation (if I'm not using it)?

    Name:  memory.jpg
Views: 45
Size:  29.9 KB

    Thanks,

    Andy

  2. #2
    Join Date
    Jan 2015
    Posts
    107

    Default

    To me that seems fairly normal and even desirable. Having more of the data in memory speeds up the processing. 2 GB out of 8 GB is not that bad either, and your total mem use seems to sit at 75% only.

    That said, you should pay attention to which jobs and transformations cause the memory use to grow.

    • Certain steps try to load all of their (lookup) rows into memory.
    • Copying rows to (job) results means storing them in memory. Take care with large datasets and clear the results at the first opportunity.
    • Looping in a job may cause increased memory use too in my experience, among other risks

  3. #3
    Join Date
    Dec 2016
    Posts
    27

    Default

    Question - How do you clear results for an ad-hoc transformation?

    Thanks,

    Andy

  4. #4
    Join Date
    Jan 2015
    Posts
    107

    Default

    After finishing execution of a transformation, the engine should release all its memory. That memory will be available to the application for reuse, so you will not see that in the performance monitor until Garbage Collection runs in the JVM, which may or may not decide to shrink the heap.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.