Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Incremental Load in Pentaho

  1. #1
    Join Date
    Jul 2015
    Posts
    5

    Talking Incremental Load in Pentaho

    Hi

    Currently i am doing Truncate and load on my target.Instead of this i want to incremental load(take only new/updated record).I used insert/update transformation It is taking around 17 hours to complete.Can you please tell me how to do this in pentaho.I am from Informatica background i know how to achieve this in Informatica.what is the equivalent transformation in pentaho.How to do it in pentaho.

    Thanks
    Shekar

  2. #2
    Join Date
    Mar 2015
    Posts
    180

    Default

    you can use audit column for your incremental records.

    if your job is taking 17 hours means, where are you running your job, is it in your local machine or on server? Where is your source and Target database located??

    is indexes created on your comparison id or key in insert/update step ? alternative for insert/update step is Sync after merge step. you can upload your transformation then we can help you mostly.

  3. #3
    Join Date
    Mar 2014
    Posts
    181

    Default

    Quote Originally Posted by ranala View Post
    you can use audit column for your incremental records.

    if your job is taking 17 hours means, where are you running your job, is it in your local machine or on server? Where is your source and Target database located??

    is indexes created on your comparison id or key in insert/update step ? alternative for insert/update step is Sync after merge step. you can upload your transformation then we can help you mostly.

    In case you need help, please try to give as much detailed information as you can e.g how many records of data you are reading, the database you are using, plus the environment you are running kettle from.

    For starters:

    Look at the Merge rows mergs 2 streams of data and add a flag example transformation , it comes with pdi in the sample directory.

    Replace the text file input step with the table input step in both streams.

    Also before the Merge join step , sort your data on join keys in ascending order.

    Lastly replace the text file output step with the synchronize after merge step.

    Hopefully that helps.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2017 Pentaho Corporation. All Rights Reserved.