Hitachi Vantara Pentaho Community Forums
Results 1 to 9 of 9

Thread: Job statistic logging to a webservice

  1. #1
    Join Date
    Dec 2011
    Posts
    5

    Default Job statistic logging to a webservice

    I would like to log the statistics of a job to an external webservice that we use to log various information in our oganization.

    Ideally this would allow me to pass the statistics of the job (inserts, updates, deletes, errors).

    It doesn't look like this is supported oujt of the box, am I missing it, or are there any suggestions on how I could accomplish this?

    Thanks,
    Jonathan

  2. #2
    Join Date
    Sep 2009
    Posts
    810

    Default

    Hey there,

    would it be an option for you to enable database logging for your jobs, and then have a follow-up job/transformation in place that just pulled the information from there and post it to the webservice?

    Cheers
    Slawo

  3. #3
    Join Date
    Dec 2011
    Posts
    5

    Default

    That would be an option. Pentaho is being used as part of a BMC Atrium CMDB project. I believe stats are already logged to the BMC database. Is there a way to *also* log to a custom DB, or in general how can I get the job statistics to log directly with SQL?

  4. #4
    Join Date
    Sep 2009
    Posts
    810

    Default

    Just take a look at the logging tab of the job settings dialog (Ctrl+J for jobs, Ctrl+T for Transformations). DB logging is defined there. If it is enabled, you just go to the tables that are logged to and query them directly with SQL to get the stats. If logging is not enabled, just supply a connection, define a table name, and have the job/transformation log to that.

    Cheers
    Slawo

  5. #5
    Join Date
    Dec 2011
    Posts
    5

    Default

    Thanks for pointing me to the logging settings.

    I think I need to add another job at the end of my sub jobs that is an SQL script, this will pull the current jobs statistics from the logging database, and call a stored proc in another DB to logg my statistics to another system. Whgat variables are avialable to select the current jobs data from the logging table? Can you point me to any available documentation on job variables that are available?

  6. #6
    Join Date
    Sep 2009
    Posts
    810

    Default

    Best thing would be to give you a short demo really. I'd create a MySQL or Postgres based sample for you if that would be ok.

    Cheers
    Slawo

  7. #7
    Join Date
    Dec 2011
    Posts
    5

    Default

    That would be greatly appreciated.

  8. #8
    Join Date
    Sep 2009
    Posts
    810

    Default

    Hey There,

    attaching a sample that may show the idea. I got logging set up (nonstandard mysql port, beware) and have a master job that

    1. executes a worker job
    2. pulls that worker job's info out of the log tables (including sub-transformations and subjobs it might have run)

    The worker job needs to somehow let the outside world know its batch id (generated for each run). I do that at the end of the worker using a short script and set variables.

    Hope it works for ya. Small SQL dump of the logging tables attached.

    Cheers
    Slawo
    Attached Files Attached Files

  9. #9
    Join Date
    Dec 2011
    Posts
    5

    Default

    Any ideas on how to call a webservice from kettle? I want to post data to the webservice, not retrieve it.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.