Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Send email if job takes longer than usual to execute - is it possible?

  1. #1
    Join Date
    Aug 2008
    Posts
    563

    Default Send email if job takes longer than usual to execute - is it possible?

    Hi,
    For me it's quite important to monitor the performance of my jobs. It happens that we have data explosions from time to time and hence the job would run longer. On this occasions I'd like to be emailed that my job is running longer than expected. Ideally this would be based on an average of the duration of the last x days.
    Is this possible?
    Best regards,
    Diethard
    ===============
    Visit my Pentaho blog which offers some tutorials mainly on Kettle, Report Designer and Mondrian
    ===============

  2. #2
    Join Date
    Dec 2009
    Posts
    332

    Default

    I have configured a job that does exactly what you are describing, but perhaps not using the best of what is available within Pentaho to do it. My choice was to have two separate jobs which start up at the same time. One is the long running process which I want to monitor and the other is the monitor process. The main job inserts a new record into a table when it initiates and updates that record when it completes. (I had originally planned to use the PDI log tables but that did not work out well.) The monitor job waits for the new record, then calculates the historical average based on the successfully completed reocrds. The monitoring job then waits the average time plus 10% and checks again. If the main job has not completed, it sends out an email and starts waiting 10% of the total time again before checking again. The monitoring job sends out continuous messages until it sees a completion or a failure or reaches a run time that I assume is a failure.

    I would imagine you could accomplish the same thing without a database, but it sure makes it simple. And while Pentaho probably has something much shinier to monitor things, this method has few moving parts and is reasonably reliable.

    Does this make sense?

  3. #3
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Makes a lot of sense. Jens also described an even more advanced watchdog concept for transformations and jobs on his blog.

    Personally I wouldn't make things all that complicated. Simply build a list of min and max execution times for your jobs and use that to send warning mails. That should work fine.
    Please note you don't have to wait until the job finishes to sound the alarm. You can do interval logging these days so you can check progress.

    Matt

  4. #4
    Join Date
    Aug 2008
    Posts
    563

    Default

    Thanks a lot both for your feeback!

    @Matt: If I use interval logging, I would still have to create a second job which checks on the main job or could this be done all in one?

    I guess I would have to create a generic job which does the checking and include it with all the normal jobs.
    Best regards,
    Diethard
    ===============
    Visit my Pentaho blog which offers some tutorials mainly on Kettle, Report Designer and Mondrian
    ===============

  5. #5
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    I would create a separately scheduled job that checks all the other jobs. That's the main design principle behind the watchdog concept.

    For 4.3.0 I have planned to allow for job entries to time out and fail after a certain maximum time but that's not always the solution. It's always possible a job is exceptionally slow because of a slow DB, network and so on. Killing the job would make things worse, not better. In other cases when the job gets stuck in a db deadlock or something of that nature it's better to call it a day.

  6. #6
    Join Date
    Aug 2008
    Posts
    563

    Default

    Ok, thanks a lot!
    Best regards,
    Diethard
    ===============
    Visit my Pentaho blog which offers some tutorials mainly on Kettle, Report Designer and Mondrian
    ===============

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.