Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Transformation time limits?

  1. #1
    Join Date
    Dec 2009
    Posts
    332

    Default Transformation time limits?

    This morning our main ETL process was waiting patiently for 3 hours on a table lock to perform a truncate. Let's just say that this was not preferable. Unfortunately since the job/transformation had not "failed", no alert messages were sent. Similar issues have occured in the past where a database lock was not the cause (network connections, etc). We would prefer to have the job fail after a defined period of time for each transformation.

    We are in the process this morning of designing external monitoring for this type of event, but it seems like something that might or should be included in Pentaho.

    Is there a method within a job to timeout transformations if they exceed a time limit or some other way to resolve this type of issue within kitchen?

    Thanks.

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Not yet but I've been thinking about implementing it in v4.2.

    On the subject, Jens Bleuel gave a Watchdog presentation at the community event in Lisbon.
    The idea is that you increase a counter in a table somewhere.
    On a higher level this idea already works with standard logging in PDI. Simply check the logging table for transformations that have status "start" or "running" with another independent transformation. Those will be the ones stuck or crashed.

    Matt

  3. #3
    Join Date
    Dec 2009
    Posts
    332

    Default

    Thank you for the quick reply. Looking forward to a possible implementation.

    In the meantime we have implemented some monitoring of the logging tables using the 'start' and 'end' statuses.

    Also, since we have several months of history of these same jobs running, we have created a rolling average expected completion time for each job, transformation and step. If any of these fail to meet their expected completion times by a large enough margin, it triggers an alert. Unfortunately the response to these alerts is manual, but it seems like a good start.

    Thanks again for your help.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.