Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Kitchen startup slow and huge memory footprint

  1. #1
    Join Date
    Aug 2009
    Posts
    2

    Question Kitchen startup slow and huge memory footprint

    Hello,

    I'm new to the kettle community but have been using it with Spoon for the several months. I'm using the 3.2_stable verion and have some weird behavior when I deploy from the desktop using Spoon to a linux server using kitchen.

    My questions about configuring for use with kitchen and pan on a linux server. I've created several jobs/transformations that run very quickly on my local windows environment in Spoon. When I execute them on a linux server with Kitchen they have a very slow startup. It seems that kitchen is loading up all of the jars each time I execute the job. So, startup is about 15 seconds and sucks about 800Mb of memory. I hope to reduce this overall startup and footprint.

    I've poured over the doc and have seen references to .kettle directories and kettle.properties files but no examples or complete doc. Is that the right path to configure this?

    Also, has the dev team considered making Kettle a service that can be called on without a startup each time? That would be great for the use in a SOA environment where we could just send JMS messages to Kettle and get results back to message bus.

    Thanks for any help. Like I said I think this thing is great. Sorry for asking the basic questions but I feel like I missed something obvious to all the other users.

    Bill

  2. #2

    Default

    In the .kettle directories you find information like your variable definitions in kettle.properties, your repository connections, your shared objects and your saved kettle options. So, this is very likely not the place you are looking for.
    As you have probably used them by now, the <name>.sh files are the Unix start scripts for the various programs. One thing you might want to configure is the Xmx option. Which is set in spoon.sh to only 256m. You might want to set that to 1500m if you are running 32bit Linux or something higher if you are using a 64bit Linux and use a 64bit Java. Since I have no clue, what your job and transformations look like, I have no idea, why it should use 800MB.

  3. #3
    Join Date
    Aug 2009
    Posts
    2

    Default

    Thanks for the reply. I don't see a .kettle directory or kettle.properties file on my installation nor could I find any doc on these. Do I have a bad install?

    Regarding the memory use, it seems that kitchen is loading all the jars into memory rather than just those used in the transformation.

  4. #4
    Join Date
    Feb 2009
    Posts
    296

    Default

    Thanks for the reply. I don't see a .kettle directory or kettle.properties file on my installation nor could I find any doc on these. Do I have a bad install?
    The .kettle directory is in the user's home dir. If you're running kitchen in your own user try
    Code:
    ls -la ~/.kettle/
    - it should show you the kettle.properties file.
    The documentation should be in the data-integration directory. However this depends on the way you've installed PDI. I've had to create a RPM of PDI and stripped it of documentation and samples (mainly because of pesty space characters in file and directory names but also due to their size). Just check any install - there should be docs. However - they are no longer up to date - you should rather try the wiki.

    Regarding the memory use, it seems that kitchen is loading all the jars into memory rather than just those used in the transformation.
    The kitchen.sh script is adding each and all of the jars within the PDI lib directory to the classpath. I don't really know how your JRE handles this but it might very well mean that it's loading them all up.
    Fabian,
    doing ETL with his hands bound on his back

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.