Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Monitoring and changing application state (jobs / transformations)

  1. #1
    sven.thiergen Guest

    Default Monitoring and changing application state (jobs / transformations)

    Some notes on the monitor topic, as I did not find anything else,
    regarding this:

    Has there ever been the demand to monitor the state of the currently
    active jobs and transformations in a production environment? We
    currently have a request from a customer of some data extraction/
    transformation which could be handled very well with Kettle.

    However, he wants additionally to able to
    .. monitor the current state of Jobs
    -> did they fail at a certain time?
    -> what was the reason of the failure?
    .. change job / transformation settings, if something failed

    And all this via web interface. As I understand, jobs and
    transformations can only be monitored and set up via the application
    itself. Is it a good idea to create some kind of TCP/IP interface,
    which could be used to request the currently running jobs,
    transformation etc.?

    If a repository gets used, how many information about execution gets
    logged in the tables? E.g. execution time, failure reasons etc.?


    Regards,
    Sven.


    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

  2. #2
    Matt Casters Guest

    Default Re: Monitoring and changing application state (jobs / transformations)

    Hi Sven,

    This is another interesting topic you raise.
    We are actually designing a managment system, API if you will, not
    just for Kettle but for the whole Pentaho suite of products.

    That being said, we don't have to wait for this.
    You can already interrogate the state of a running transformation on a
    slave server, start it, stop it. We will doing the same thing for
    jobs once 3.0 porting is done. I guess it will be in 3.0M2 or M3.
    Any help is welcome, both on ideas and on implementation.

    I have a strong liking for using HTTP requests for this type of usage.
    It provides a clear contract with the clients and it allows for
    easier debugging. On top of that it's completely language independend.

    All the best,

    Matt


    Quoting "sven.thiergen" <s.thiergen (AT) itcampus (DOT) de>:

    >
    > Some notes on the monitor topic, as I did not find anything else,
    > regarding this:
    >
    > Has there ever been the demand to monitor the state of the currently
    > active jobs and transformations in a production environment? We
    > currently have a request from a customer of some data extraction/
    > transformation which could be handled very well with Kettle.
    >
    > However, he wants additionally to able to
    > . monitor the current state of Jobs
    > -> did they fail at a certain time?
    > -> what was the reason of the failure?
    > . change job / transformation settings, if something failed
    >
    > And all this via web interface. As I understand, jobs and
    > transformations can only be monitored and set up via the application
    > itself. Is it a good idea to create some kind of TCP/IP interface,
    > which could be used to request the currently running jobs,
    > transformation etc.?
    >
    > If a repository gets used, how many information about execution gets
    > logged in the tables? E.g. execution time, failure reasons etc.?
    >
    >
    > Regards,
    > Sven.
    >
    >
    > >

    >
    >




    -------
    Matt Casters
    Chief Data Integration
    Pentaho Open Source Business Intelligence


    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

  3. #3
    sven.thiergen Guest

    Default Re: Monitoring and changing application state (jobs / transformations)

    Well, I get the idea (started Carte, controlled it all within my web
    browser) ... this all looks very much like what we had in mind. HTTP
    requests are good for the mentioned reasons. What do you think about a
    pure data interface instead of returning the data packed within
    (simple) HTML? Customers often will ask for a web page of their
    liking, so pure data will be useful in that case.

    I am not that experienced in loading whole transformations from a
    repository or XML (right now I've seen the programmatic creation of
    one transformation "row generator test") but somehow it will be
    possible. Same for jobs.

    One could even provide access to transformation properties and change
    them via Web interface (e.g. change the filename of a csv file via web
    interface, leaving anything else as it is). What do you think about
    that? Is it a useful thing or rather too dangerous, when people are
    able to change anything from anywhere?


    All the best,
    Sven.


    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

  4. #4
    Matt Casters Guest

    Default RE: Monitoring and changing application state (jobs / transformations)

    Hi Sven,

    Indeed, an HTML web-interface is not what's typicall being used by
    applications.
    Because of that we have an xml=Y option that you can use to have the output
    sent to you in XML.

    I hope that this answers your question. Personally I think XML is a lot
    better than CSV for these things, but if you really want to add another
    csv=Y option, be my guest :-)

    All the best,
    Matt

    -----Original Message-----
    From: kettle-developers (AT) googlegroups (DOT) com
    [mailto:kettle-developers (AT) googlegroups (DOT) com] On Behalf Of sven.thiergen
    Sent: Wednesday, May 30, 2007 3:12 PM
    To: kettle-developers
    Subject: Re: Monitoring and changing application state (jobs /
    transformations)


    Well, I get the idea (started Carte, controlled it all within my web
    browser) ... this all looks very much like what we had in mind. HTTP
    requests are good for the mentioned reasons. What do you think about a pure
    data interface instead of returning the data packed within
    (simple) HTML? Customers often will ask for a web page of their liking, so
    pure data will be useful in that case.

    I am not that experienced in loading whole transformations from a repository
    or XML (right now I've seen the programmatic creation of one transformation
    "row generator test") but somehow it will be possible. Same for jobs.

    One could even provide access to transformation properties and change them
    via Web interface (e.g. change the filename of a csv file via web interface,
    leaving anything else as it is). What do you think about that? Is it a
    useful thing or rather too dangerous, when people are able to change
    anything from anywhere?


    All the best,
    Sven.





    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

  5. #5
    Matt Casters Guest

    Default RE: Monitoring and changing application state (jobs / transformations)

    Before I forget, we have objects that can be created from the XML that comes
    back.
    So if you have kettle.jar in your classpath you should not worry about the
    XML.
    For example StepStatus(Node node), SlaveServerTransStatus, etc.

    Matt


    -----Original Message-----
    From: kettle-developers (AT) googlegroups (DOT) com
    [mailto:kettle-developers (AT) googlegroups (DOT) com] On Behalf Of Matt Casters
    Sent: Wednesday, May 30, 2007 3:48 PM
    To: kettle-developers (AT) googlegroups (DOT) com
    Subject: RE: Monitoring and changing application state (jobs /
    transformations)


    Hi Sven,

    Indeed, an HTML web-interface is not what's typicall being used by
    applications.
    Because of that we have an xml=Y option that you can use to have the output
    sent to you in XML.

    I hope that this answers your question. Personally I think XML is a lot
    better than CSV for these things, but if you really want to add another
    csv=Y option, be my guest :-)

    All the best,
    Matt

    -----Original Message-----
    From: kettle-developers (AT) googlegroups (DOT) com
    [mailto:kettle-developers (AT) googlegroups (DOT) com] On Behalf Of sven.thiergen
    Sent: Wednesday, May 30, 2007 3:12 PM
    To: kettle-developers
    Subject: Re: Monitoring and changing application state (jobs /
    transformations)


    Well, I get the idea (started Carte, controlled it all within my web
    browser) ... this all looks very much like what we had in mind. HTTP
    requests are good for the mentioned reasons. What do you think about a pure
    data interface instead of returning the data packed within
    (simple) HTML? Customers often will ask for a web page of their liking, so
    pure data will be useful in that case.

    I am not that experienced in loading whole transformations from a repository
    or XML (right now I've seen the programmatic creation of one transformation
    "row generator test") but somehow it will be possible. Same for jobs.

    One could even provide access to transformation properties and change them
    via Web interface (e.g. change the filename of a csv file via web interface,
    leaving anything else as it is). What do you think about that? Is it a
    useful thing or rather too dangerous, when people are able to change
    anything from anywhere?


    All the best,
    Sven.








    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.