Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Missing Dimension Records in Longer running ETL

  1. #1

    Default Missing Dimension Records in Longer running ETL

    me again... lol

    created an ETL ran perfectly, the number of records for a specific dimension was exactly == to the number of bottom level items for that dimension in the facts table.

    now i modified it and due to certain process added, it takes much longer to run, however i didn't not change anything in relation to the update of the dimension. after execution, i found that the number of records within the dimension tables were much less than they were supposed to be.

    any idea or solutions?

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    That's pretty vague Sean. So you think that we somehow mysteriously delete records from a table? LOL
    Mmm, perhaps you have another connection open and forgot to do a commit or something? That can play tricks with you.

    Matt

  3. #3

    Default

    err... i dunno ... u tell me .....lol


    in that case i'll try and adjust the commit size and see if that makes any difference

    another question.... which ETL design method (if there is such a thing) is better.... serial or parallel.... in relation to the writing records to dimensions and fact table that is

    currently i have processes relating to each dimension and fact branching off from common information and running in parallel... whilest in the "Building data warehouses usingopen source technologies" pdf in docs folder i see the approach being used to be some what serial

    i wonder if that cud be the source of my problem... hmmmm
    Last edited by smilez2k7; 03-27-2008 at 05:34 PM.

  4. #4

    Default

    i ran the biggest transformation of the job alone... and realized that the dimension that was updated with the "few" records, was the only one that was updated @ all

    in addition, i realise that all the objects on "branch" with the facts table on it, got all the info it shud, while all the other branches... "starved" so 2 speak

    serial FTW?

  5. #5

    Default

    this is what i meant by parallel.... cud the design be the cause?
    Attached Images Attached Images  

  6. #6
    DEinspanjer Guest

    Default

    Quote Originally Posted by smilez2k7 View Post
    this is what i meant by parallel.... cud the design be the cause?
    How do the new dimension entries you create in this process ever get their technical keys inserted into the fact table as foreign keys? This seems very odd to me.

  7. #7

    Default

    Quote Originally Posted by DEinspanjer View Post
    How do the new dimension entries you create in this process ever get their technical keys inserted into the fact table as foreign keys? This seems very odd to me.

    those keys are added to the facts table data stream via the "derive terminal type" and "add product" objects

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.