Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: CDC logging for looping through different dbs

  1. #1
    Join Date
    Dec 2017
    Posts
    2

    Default CDC logging for looping through different dbs

    I have a job which I need to loop through the same set of transformations for every site (same db schema across the country) for which I'm bringing data into a data warehouse. Everything is done dynamically at job level to make sure that connections and other pieces of config are switched to the next site every time one is finished until I run through all of them.

    The only thing I cannot change are the transformation names, because they won't accept variables. The problem here is that I absolutely love and depend on the out of the box logs for the built-in CDC feature to work, and it uses the transformation names to keep track of star/end dates. I have to record this at transformation + site level, because they will sometimes run at different times of the day (in case of comms having been down, for example).

    The only way I could find around this was physically creating copies of the whole set of transformations for each site, manually renaming them (adding site name) and running one job for each site. But this makes maintenance/enhancements a pain in the butt, because it means applying that tiny small change 10 times. Deployment to QA and Prod is also very painful.

    Does anyone have ever gone through this same problem? Suggestions?

    Also, it will be great that experts can give good examples.

  2. #2
    Join Date
    Dec 2017
    Posts
    2

    Default

    one more question: if we have id column too (with auto increment number), is it better idea to use it as the key for CDC instead of using date_time column?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.