Hitachi Vantara Pentaho Community Forums
Results 1 to 8 of 8

Thread: Dimension update

  1. #1
    Join Date
    Nov 2007
    Posts
    5

    Default Dimension update

    Hello all,

    I have this dilemma regarding the update of dimension entries. Sometimes you have fields that can either be updated (resulting in a new version), or corrected (resulting in an update of the existing version), but there is no way to distinguish between both cases in the source data.

    Now I want to 'solve' this problem in the following way:
    If the dimension entry is recent, say less than a year, then the entry should be updated, if the entry is older, a new version should be created.

    If you use Kettle's dimension lookup/update step, it is not possible to see how old a dimension entry is. What would be the easiest way to do this?
    All suggestions are welcome.

    Thanks in advance,
    Axel Hallez

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    Find another way... what you want to do is more or less guess at what should be performed which is not in the dimension step, and will probably never make it in there

    If you want to do something like that you would already have too lookup the data of the last row beforehand, compare it, and adapt the original... all of which will become very slow.

    Regards,
    Sven

  3. #3
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    If you don't know which of the 2 you need, my advice is to include both in the dimension entries.
    For example, if you have a "name", you can include "name_hist" with an Insert setting (for historically correct) and "name_last" (for last version applied) with a "Punch through" setting.
    The "Update" setting should only be used to correct the last version of a dimension entry and perhaps to update the last changed date or something like that.

    Matt

  4. #4
    Join Date
    Nov 2007
    Posts
    5

    Default

    Let me explain my problem with an example. I have a Person dimension that includes address data. The Persons are members of sports clubs. In the source database the data is entered by clubs administrators.

    Now, if an address changes in the source database, this can be for two reasons:
    - The address was entered was wrong and is now corrected. In this case the dimension entry versions using the wrong address should be corrected.
    - The person has moved. In this case a new version should be made.

    However when somebody changes an address in the source database, the reason for the change is not recorded. However, my guess is that if an address is changed shortly after the person is created, the change is probably a correction, after a while, changes are more likely to be a genuine change.

    That is why I wanted to know if there is a way to implements this.

    Kind regards,
    Axel Hallez

  5. #5
    Join Date
    May 2006
    Posts
    4,882

    Default

    Refer to my first reply ... Doable, but not that easy (read: not already built in somewhere), and not recommended.

    Regards,
    Sven

  6. #6
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Axel, for once I'm actually tempted to consider this a genuine feature request. Certainly it's valid to say that if a person moves from one address to another, he or she is not going to do this every 2 weeks or so.
    It would be cool if you could make the condition customizable, with an expression of some sort. It's worth some consideration.

    Make sure to file a feature request for it. Who knows what happens in 2009

    Matt

  7. #7
    Join Date
    May 2006
    Posts
    4,882

    Default

    Lol ... I did similar solutions... and then 1 day you hit a row which you throw away as being not relevant while the row actually was relevant.

    I think it's pretty hard to make something general for it.

    Regards,
    Sven

  8. #8
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Did I say 2009? Aw, that's not what I meant at all ;-)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.