Hitachi Vantara Pentaho Community Forums
Results 1 to 8 of 8

Thread: Handling Idle connections within the flow

  1. #1

    Default Handling Idle connections within the flow

    Hi friends!

    I got some connections that sometimes get hung because of the network or something. As long as we don't identify the concrete problem and fix, I am wondering if there is any sort of Timeout for handling an idle connection. I'd like to abort the job and follow with the cancel flow in case the transformation surpasses a time limit.

    Is that possible?

    Thank you!

  2. #2

    Default

    The "wait for" step is maybe what you are looking for. Its in the job level.

  3. #3

    Default

    Quote Originally Posted by smartysidster View Post
    The "wait for" step is maybe what you are looking for. Its in the job level.
    Hi ! Thanks!

    Do you mean having a parallel flow sleeping during some time and then use it for aborting the job?
    Mmh, that's an option but if so I think it's a bit annoying implementing that for every job...

    Perhaps any other jdbc parameter or trick or default setting in kettle?

  4. #4
    Join Date
    Sep 2013
    Posts
    235

    Default

    Just a suggestion.
    If your jobs transformations will run on EE DI server consider to use JNDI - and see what context.xml can offer for this case.
    This is typicall JNDI declarations I took from default installation:

    Code:
      <Resource name="jdbc/Audit" auth="Container" type="javax.sql.DataSource"
                factory="org.apache.commons.dbcp.BasicDataSourceFactory" maxActive="20" maxIdle="5"
                maxWait="10000" username="hibuser" password="password"
                driverClassName="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/di_hibernate"
                validationQuery="select 1" />
    I would try to play with maxWait parameter here. But did not try it in real world. Will it help? I don't know for your case.

  5. #5

    Default

    Quote Originally Posted by Dzmitry View Post
    Just a suggestion.
    If your jobs transformations will run on EE DI server consider to use JNDI - and see what context.xml can offer for this case.
    This is typicall JNDI declarations I took from default installation:

    Code:
      <Resource name="jdbc/Audit" auth="Container" type="javax.sql.DataSource"
                factory="org.apache.commons.dbcp.BasicDataSourceFactory" maxActive="20" maxIdle="5"
                maxWait="10000" username="hibuser" password="password"
                driverClassName="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/di_hibernate"
                validationQuery="select 1" />
    I would try to play with maxWait parameter here. But did not try it in real world. Will it help? I don't know for your case.

    Nice feature! unfortunately we don't use EE DI Server :'(. Thanks anyway! Hope I'll use it someday!

  6. #6
    Join Date
    Sep 2013
    Posts
    235

    Default

    Quote Originally Posted by bizintreader View Post
    Nice feature! unfortunately we don't use EE DI Server :'(. Thanks anyway! Hope I'll use it someday!
    If you use CE and database repository you may notice that all connections is stored centralized and referenced by connection name,
    so try to edit connection property with repository browser, select Options/Parameters, and try to set parameter, for mysql that could be 'wait_timeout' or something else.

    That is only for db connection, for example ftp connection can specify wait timeout for their steps directly. That may (or may not) help too and that will be CE recipe.

  7. #7

    Default

    I'll try that way, experimenting with it, thanks!

  8. #8
    Join Date
    Apr 2008
    Posts
    4,689

    Default

    Usually, you will see these problems with MySQL.
    The other DB engines seem to handle a reconnect gracefully.

    The traditional solution is to put 3cp0 between your DI and the MySQL DB. This is handle the reconnect for you.

    Since the DB connection is established when the Transform initializes, it can take some time for the data to come from the source data, be processed, etc, before it hits a later step which uses the DB. If this takes too long, the DB would timeout, and need to have a reconnect.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.