Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Kettle Quesion !!

  1. #1

    Default Kettle Quesion !!

    Hi Matt

    Recently I have done POC on KETTLE and Presented usage of Kettle to executives which was really impressive.

    And I have question which I need answers myself and to a lead. (I am purely new to ETL)

    We are product based company and we build software on top of ERP namely develop SOX compliance software.

    Here are question we have.

    • How can I call the jobs or transformations from command line – Would be using Pan or kitchen (documentation or sample would help me). Is there any way that I can create executable scripts in for UNIX ,Linux etc
    • How do I package the kettle with our product and our application is shrink wrapped application and deploy it in various customer sites
    • How big is the foot print?
    • how do we monitor job and transformation offline and how can we keep track of records processing
    • Is there any java API to run the transformation and jobs
    • Right now we developed few Transformations and jobs. How do I deploy them in QA instance, how do I change the connection information dynamically (I don’t want to share the use Shared Connection and but I should be able to create new connection while I am deploying the transformation and jobs dynamically) or you can suggest best way approach.
    • How can we use spoon on Linux and UNIX. Is there way to create a .sh file which executes the transformations or jobs?

    Issue while using kettle.

    • I have created four transformation which populates 45000 records in one table and If I execute each transformation I don’t have any problem but if I create the job including all 4 transformation and run it on GUI , spoon designer window terminates session automatically.
    • Installing spoon on UNIX is error out. How do I set the class path?

    Does kettle provide technical support and how do we establish communication?

    Matt- I have mailed you to your mail and i am wait for response.

    Thanks
    Regards

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Lightbulb

    How can I call the jobs or transformations from command line – Would be using Pan or kitchen (documentation or sample would help me). Is there any way that I can create executable scripts in for UNIX ,Linux etc
    Documentation for both Pan and Kitchen, including use-cases and scheduling samples are in the docs/English directory.

    How do I package the kettle with our product and our application is shrink wrapped application and deploy it in various customer sites
    I think only you can answer that question as it highly depends on how you want to package it.

    How big is the foot print?
    Again, that depends on what you want to support, which databases, platforms, etc. I think if you only keep the minimal libraries you need for 1 database and one GUI platform, you would probably end up with less than 30MB

    how do we monitor job and transformation offline and how can we keep track of records processing
    Please see our technical tip on auditing, in the docs/English/tips directory

    Is there any java API to run the transformation and jobs
    Yes there is. In fact the source code for Pan and Kitchen is available and not too complex. You can even define new transformations using the API, see here .

    Right now we developed few Transformations and jobs. How do I deploy them in QA instance, how do I change the connection information dynamically (I don’t want to share the use Shared Connection and but I should be able to create new connection while I am deploying the transformation and jobs dynamically) or you can suggest best way approach.
    This is easily achieved through the use of variables. See the Spoon documentation chapter on variables.
    The gist of the story is that you can set all variables on the system in $HOME/.kettle/kettle.properties or $KETTLE_HOME/.kettle/kettle.properties
    That way, you can create connections that use the variables and not fixed parameters. The values can then be changed from DEV to TEST to PRODUCTION

    How can we use spoon on Linux and UNIX. Is there way to create a .sh file which executes the transformations or jobs?
    Simply executing this will do the trick:

    Code:
    sh spoon.sh
    I have created four transformation which populates 45000 records in one table and If I execute each transformation I don’t have any problem but if I create the job including all 4 transformation and run it on GUI , spoon designer window terminates session automatically.
    It's probably an out of memory error that you get, but without a trace it would be difficult to say. Try increasing the amount of memory available to Spoon in Spoon.bat/spoon.sh indicated with the Xmx Java parameter.

    Installing spoon on UNIX is error out. How do I set the class path?
    The classpath on Linux/AIX/HPUX/OSX and Solaris is set automatically. We don't support Spoon on platforms other than that.

    Does kettle provide technical support and how do we establish communication?
    We offer excellent technical support, training classes and services at a fair price. Someone will contact you shortly to establish communcations on that subject.

    All the best,

    Matt

  3. #3

    Default

    Thanks for your response, Matt.

    I was able to easily execute Spoon transformations from the command line. I have few more questions:

    1. Can you provide me examples of using Pan to execute jobs?

    2. Also examples of using Spoon to execute transformations and jobs from the command line.

    3. In your response you had mentioned that you set the db connection properties in kettle.properties file. Can you provide the exact property names? Please also point me at the doc that describes these properties. Will Pan also use this property file for executing transformations and jobs?

    4. It seems like I can pass transformations and jobs as arguments to spoon and execute them, is this true? Is so, what is the difference between using this and pan?

    Thanks

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    1. Can you provide me examples of using Pan to execute jobs?
    Pan is used to execute transformation, documentation with examples is here: docs/English/Pan-2.4.pdf
    Kitchen is used to execute transformation, documentation with examples is here: docs/English/Kitchen-2.4.pdf

    2. Also examples of using Spoon to execute transformations and jobs from the command line.
    That is obviously not a good idea and not possible :-)

    In your response you had mentioned that you set the db connection properties in kettle.properties file. Can you provide the exact property names? Please also point me at the doc that describes these properties. Will Pan also use this property file for executing transformations and jobs?
    You can use any property names you like. Documentation is in docs/English/Spoon-2.4.pdf (chapter 9 : Variables)
    Pan & kitchen can and will indeed also use these properties.

    It seems like I can pass transformations and jobs as arguments to spoon and execute them, is this true? Is so, what is the difference between using this and pan?
    Spoon is a graphical user interface and as such not suitable for batch execution of transformations and jobs outside of development, testing and debugging. In any batch run it is best to make everything with as little (none, zero) user interaction as possible.

    HTH,

    Matt

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.