Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Any Way to Limit Kettle Job Memory Usage?

  1. #1

    Default Any Way to Limit Kettle Job Memory Usage?

    Hi,

    Is there a way to limit the amount of memory a Kettle job uses? I have a fairly large job, and it is consuming nearly 1Gb of memory. I would rather it use less memory even if it would run slower.

    If the amount of memory cannot be limited, are there operations, perhaps logging, that are known for consuming a large amount of memory that I could turn off?

    Thanks for any help.

    Stuart

  2. #2
    Join Date
    Mar 2011
    Posts
    29

    Default

    Related:

    I've configured (in Spoon.bat) -Xmx1024m and have run a transformation many times on a Pentium-4 and a Core-2. Now, I moved to a new Core-i7 and run out of memory. (Heap overflow). Going to -Xmx2048 fixed the issue.

    Question: Is it reasonable to assume that Kettle parallelizes more on the 8 cores and a consistent 64-bit installation than before, hence, consumes more memory?

    (Edit: Maybe I should be a little more detailed: Transformation consists of nothing special:
    - loading an .xls
    - filter out some rows
    - trimming some fields
    - selecting some fields
    - loading into a mySQL staging table)

    Stefan
    Last edited by Stefan_E; 01-14-2012 at 12:08 PM.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.