Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Resume a transformation

  1. #1
    Join Date
    Aug 2010
    Posts
    29

    Default Resume a transformation

    Hello everyone,

    I'm new in all this whole datawarehousing topics but first I'd like to thank you all for this huge commmunity effort.

    My question is the following.

    I have a source database, transformation and a target database and I want to run the ETL process monthly. when I run it for the first time, I extract all the data then I transform it and load it to the target database without problems.

    When I run the ETL for the second time I have that all the data from the first etl process is now duplicated in the target database. I have the option to truncate the table before each transform but this option will make the ETL process take each time longer.

    Is there any way to make a transformation resume from where the last transformation left?
    Do I make my self clear?

    Thanks all in advance

    Lophophora

  2. #2
    Join Date
    Mar 2010
    Posts
    21

    Default One of many ways to address your situation

    Is there a transaction date to use as an input filter?
    Is there a batch number or some way to add a date to the input records.
    can you place the new records in a separate file so that you don't have to use a filter at all.

    Think about what do you do if you process records and find out later that the input file was not correct.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.