Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Data connection problems of Kettle + Quarts in the single JVM

  1. #1
    Join Date
    Oct 2008
    Posts
    1

    Default Data connection problems of Kettle + Quarts in the single JVM

    Hello.

    I wrote a class to launch Kettle jobs, and most of the codes are same as Kitchen.java and it is called in threads. (The source file is attached) I wanted launch Kettle jobs by Quartz, because the schedules of jobs are stored in the database, and some business logics were implemented for scheduling information. To load Kettle jobs directly in the single JVM, all jar files for Kettle were deployed to WEB-INF/lib directory.

    I didnt meet problems most of jobs, but Im wondering if it is safe way to launch Kettle jobs. Especially, our team-mates are suspecting leaks of database connections of this design, loading all Kettle classes directly in the Tomcat.
    In fact, I tried to use connection pools provided by JNDI, but Kettle consumed more connections than expected, and I got exceptions of exhausted connection pools. I could not be sure that it really needed more connections or some connections were not returned.

    Does anyone have similar experiences? Is it safe to launch multiple Kettle jobs concurrently in the single JVM with JNDI or normal DB connections? I guess it is safe because Carte has a similar structure, isnt it?
    Attached Files Attached Files

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    mmmm ... KettleLoader looks A LOT like Kitchen

    For each job that you would create I would expect you to get a set of connections. And that the connections would go away once the job is finished (eventually anyway by garbage collection).

    Regards,
    Sven

  3. #3
    Join Date
    Apr 2009
    Posts
    3

    Default

    benelog,

    did you went further with the example? and how were you calling the same? I cant see any quartz specific code in KettleLoader.java. Would you mind sharing that too? And was the problem of extra connection exception sorted out, how?

    sboden,

    any plugin available for kettle-quartz integartion just like the same as of "org.pentaho.plugin.quartz.JobSchedulerComponent" in the BI suite? Is the source dode available for that? presume that too is open source like pdi, is it?

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    quartz is considered on top of PDI/Kettle... so no integration for the moment. With the BI enterprise edition some kind of scheduling is provided.

    Regards,
    Sven

  5. #5
    Join Date
    Apr 2009
    Posts
    3

    Default

    Ok, so you mean the processmanager part of the system as I see it, is basically separate and different from ETL part? Can you suggest best opensource tool/solution for that too?

    Quote Originally Posted by sboden View Post
    quartz is considered on top of PDI/Kettle... so no integration for the moment. With the BI enterprise edition some kind of scheduling is provided.

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.