Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Archiving very large parent-child tables using Kettle

  1. #1
    Join Date
    Sep 2008
    Posts
    2

    Smile Archiving very large parent-child tables using Kettle

    Hi, I am new to Kettle and am looking for advice on how to write ETL jobs using Kettle (v 3.0.4) archiving very large multi-level parent child Oracle tables through PK/FK from source to target. The archive logic is simple, i.e., starting from the root parent table, locate all dependent child table records recursively and move them from different production to an archive host.

    One little twist in the logic is that we maintain a logical-to-physical table name mapping. As a result, we'll need the ability for Kettle to perform dynamic table name lookup in the SQL at run-time for all queries. Other features we look for include the ability in Kettle to perform garbage cleanup should any failure occurs during the ETL job execution; and the ability to throttle the job depending on source & target Oracle host load. Any input on Kettle WRT to the above requirements would be greatly appreciated.

    Regards,

    - dhsieh

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Looking up the records shouldn't be a problem I would think.Doing it dynamically requires a job that defines the table-names as variables to make the process generic.That too is possible but needs a bit more tinkering. I believe there are a number of examples of that on this forum and in the /samples/jobs/ folder of 3.0.4
    Matt

  3. #3
    Join Date
    Sep 2008
    Posts
    2

    Default

    Hi Matt,

    Can you please be a little more specific? like which sample job or forum posting has the information I am looking for?

    Regards,

    - dhsieh

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Hi Donovan,

    It was just a generic answer to a generic question.
    Unfortunately, blank statements like "very large data sets" are more often than not abused on this forum. At least, now that I know the company you work for, I have a better idea what you mean by that :-)
    Let me try to reply to your email in more detail.

    All the best,
    Matt

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.