Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Dependency on field names -> Listener framework

  1. #1
    sven.thiergen Guest

    Default Dependency on field names -> Listener framework

    Hi Devs,

    maybe this issue has been raised already, however I cannot find
    anything so please point me to the thread if there is one.

    The issue: Let's say I have a number of steps in my transformation and
    have field names and anything. The steps depend on each other using
    the fields by using the same name. If I change a field name in one
    step (or delete the field altogether) the following steps do not get
    noticed and will fail on next execution. The failure is often quite
    nasty - the user does a little change here and there and suddenly the
    whole transformation crashes.

    Now some ideas for an improvement: Establish a general framework of
    listeners for events of the type "FieldNameChanged" (a step is always
    a field name listener) and event producers (again, each step is a
    potential producer). Changes of field names get spread accross all
    steps which may take appropriate actions. The event data consists of
    the old and the new name. That seems to be a solution with reasonable
    effort, however it requires additional coding in a lot of steps and
    each time a step gets enhanced or changed one may have to adapt the
    "producer" and "listener" routines.

    Another idea is to have the fields some kind of global to the
    transformation. All steps share the same fields and a change to a
    field's name thus will always affect all steps that use this field.
    This would save the effort to propagate field name changes to each
    specific step and step changes / enhancements will not lead to
    additional effort regarding field name changes. The framework covers
    that completely.

    However it is a major change in the overall framework and I don't know
    whether it works in all cases. And the effort to adapt all steps in
    the first place is certainly large.


    Regards,
    Sven.


    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

  2. #2
    Sven Boden Guest

    Default Re: Dependency on field names -> Listener framework

    Nice idea, but short term I don't see it happening... maybe something
    for v3.0 ...

    Seriously, it would be hard to get it working 100% (try imagining
    something automatic for the transformation in the attachment of
    http://www.javaforge.com/proj/tracke...e&task_id=5040
    ). I would already be happy to be able to get a report of what breaks
    when I change a database field e.g.
    What we also seem to lack is a 100% closed case of what is acceptable
    in a job/transformation and what not, e.g.
    http://www.javaforge.com/proj/tracke...e&task_id=4987
    and http://www.javaforge.com/proj/tracke...e&task_id=4991

    What they do e.g. in DataStage is put the meta-data on the hops so
    that if one sides changes, the other side changes with automatically.
    But then in some cases you also have to do something manual to
    reenable some things (so it doesn't cascade all the way through)

    SAS ETL has something like you propose that automatically does stuff,
    but sometimes it doesn't behave very well, probably also because they
    can't cover all cases 100%. And then it's up to experience to know
    when to use it and when to leave it alone, which is also far from
    intuitive.

    Regards,
    Sven

    On Mar 14, 3:38 pm, "sven.thiergen" <s.thier... (AT) itcampus (DOT) de> wrote:
    > Hi Devs,
    >
    > maybe this issue has been raised already, however I cannot find
    > anything so please point me to the thread if there is one.
    >
    > The issue: Let's say I have a number of steps in my transformation and
    > have field names and anything. The steps depend on each other using
    > the fields by using the same name. If I change a field name in one
    > step (or delete the field altogether) the following steps do not get
    > noticed and will fail on next execution. The failure is often quite
    > nasty - the user does a little change here and there and suddenly the
    > whole transformation crashes.
    >
    > Now some ideas for an improvement: Establish a general framework of
    > listeners for events of the type "FieldNameChanged" (a step is always
    > a field name listener) and event producers (again, each step is a
    > potential producer). Changes of field names get spread accross all
    > steps which may take appropriate actions. The event data consists of
    > the old and the new name. That seems to be a solution with reasonable
    > effort, however it requires additional coding in a lot of steps and
    > each time a step gets enhanced or changed one may have to adapt the
    > "producer" and "listener" routines.
    >
    > Another idea is to have the fields some kind of global to the
    > transformation. All steps share the same fields and a change to a
    > field's name thus will always affect all steps that use this field.
    > This would save the effort to propagate field name changes to each
    > specific step and step changes / enhancements will not lead to
    > additional effort regarding field name changes. The framework covers
    > that completely.
    >
    > However it is a major change in the overall framework and I don't know
    > whether it works in all cases. And the effort to adapt all steps in
    > the first place is certainly large.
    >
    > Regards,
    > Sven.



    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.