Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Memory usage and crashes

  1. #1
    Join Date
    May 2007
    Posts
    6

    Default Memory usage and crashes

    Hi,

    My kettle script initially runs slowly, gets a little less slow and then crawls to a stop. The SQL SELECT completes in about 20 seconds returning 900k rows. The rest of the script chugs along at about 120rps untill kettle crashes to desktop at about 60k rows.

    I have been looking at my memory usage(2GB on local machine) and am concerned that I have made an obvious gaff to those in the know. The MSSQL Server sits at about 650MB of phisical with a similar amount of virtual. Java at 680MB with a similar amount of virtual and javaw at about 310MB with a similar amount of virtual. The values slowly creep up until it crashes. I disabled logging and the speed increased and more rows were eventually processed but it did not complete.

    I have attached my kettle file (laughter is not constructive) I have removed all lookups in favour of an initial SQL join of several tables as this was suggested on someone else's thread. The rest of the transformation is either Filters/INSERTS/JavaScript which I thought would be fairly rapid.

    MSSQL 2000
    java version 1.6.0.0_01
    kettle 2.5.0

    I would attach some of the debug info but kettle crashes. Is it dumped anywhere in the filesystem?

    Any help greatly appreciated. Matt B
    Attached Files Attached Files

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    At first sight too many steps, maybe you can see whether enlarging the memory for the JVM would help.

    And with which error does it crash... out of memory or something else?

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.