Hitachi Vantara Pentaho Community Forums
Results 1 to 9 of 9

Thread: memory usage in PDI4

  1. #1
    Join Date
    Mar 2010
    Posts
    24

    Question memory usage in PDI4

    Hi,

    i detected a huge memory usage with PDI4 compared to PDI3.2.5 with kitchen and running my job (with many subjobs and transformations)
    (pdi-ce-4.0.1-SNAPSHOT-r14022 vs. pdi-ce-3.2.5-r13832)

    i got a java out of memory, java heap error and so i created a hprof file an looked into it with the Eclipse Memory Analyzer
    97,77 % of the memory usage is used by org.pentaho.di.core.gui.JobTracker
    most of the memory is used for the logging information

    Name:  hprof_1.jpg
Views: 79
Size:  55.8 KB


    is it possible that in version 4 no garbage collector (or something like this) is running after one job has finished his work?

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    You probably have it reversed. The JobTracker in 3.2.x was indeed leaking memory when running in Spoon.
    If you think otherwise, file a JIRA case.

  3. #3
    Join Date
    Mar 2010
    Posts
    24

    Default

    Hi Matt,

    thanks for response.
    so, i proofed it once more but i`m sorry.
    pdi4 is using more memory as the pdi 3.2.5

    i have a example for you
    with pdi kitchen 3.2.5 the memory usage is max. 114 MB
    with PDI 4 within seconds to - > 1200 MB

    pdi_demo_memory.zip

    i think it is cause of the huge amount of log-informations

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    That's why I insist on having a JIRA case and a reproduction path.
    Let me know what the case is and I'll fix it ;-)

  5. #5
    Join Date
    Nov 1999
    Posts
    9,729

  6. #6
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    I went along and applied a fix. Please try a recent build of 4.0.2 and report in the JIRA case if things are better.

  7. #7
    Join Date
    Mar 2010
    Posts
    24

    Default

    Hi Matt,

    many thanks for your checkin. Very quick response
    I will test it as soon as possible.

  8. #8
    Join Date
    Mar 2010
    Posts
    24

    Default

    Hi again,

    Test OK with pdi-ce-4.0.2-SNAPSHOT-r14093
    memory usage max. 156 MB

    Very good. Thank you

    Greetings
    Stefan

  9. #9
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    alright!

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.