Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Pentaho Data Integration and Change Data Feature.

  1. #1

    Default Pentaho Data Integration and Change Data Feature.

    Hello.

    I would like to know if Pentaho Data Integration has a "Change Data Feature" which allows to handle data in real times?
    If yes, how can i use it?
    If not, is this feature in the roadmap of Pentaho Data Integration?

    Thanks for your answers.

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    You're mixing things I think.

    If you mean change data capture, it's not in. No short term plans to put it in I think, it depends totally on the database so for every type of database something would have to be written. A poor men's CCD is in using the merge row step functionality.

    Real time functionality.... currently not in, there is a JIRA for I think. Processing data via JMS e.g. Definition of real time is a bit vague of course: if you run a job via crontab every 5 minutes (or every time the previous instance stops), is it real-time or not?

    Regards,
    Sven

  3. #3

    Default

    Sorry!
    Yes its not "Change Data Feature" but "Change Data Capture".

    I ask this because i saw that Talend has the CDC feature for some Databases ( oracle, mysql, etc..).

    Thanks for your answers.

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Talend CDC is just a set of triggers that copy the data to temporary tables when it changes.
    It's not really CDC, just a bit of marketing. Anyone can deploy those on a source database.

    I don't really have a problem with that, you can do that with any ETL tool (including Kettle), but it's actually just an optimization of the data retrieval process (made incremental) at the expensive of the source system.

    Real CDC systems in the classical sense of the word use the database log files to see what got changed to allow for example near instantaneous database replication, etc.
    Those log files use with the exception of MySQL highly proprietary formats. This is why these "log sniffers" are usually prohibitively expensive and never open source.

    Matt

  5. #5

    Default

    Thanks very much for your answers.

    I have a better understanding!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.