Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Online Data Integration

  1. #1
    Join Date
    Mar 2009
    Posts
    137

    Default Online Data Integration

    Hi everybody,

    I asked already a question about "endless" running transformations a couple of days ago. Now I have a question about "online data integration". I'm not sure if this is the right term to use. But I was thinking about an input step to queue data from a tcp/ip socket, which gets data once in a while. And everytime something is received at the input this data is transformed and stored e.g. in a database or is send to another socket?

    Is something like this possible? Maybe somebody already wrote such an input step?

    Thanks

  2. #2
    Join Date
    Mar 2009
    Posts
    137

    Default

    Well... no answer yet ... so is it not possible or is it because of my explanation

  3. #3
    Join Date
    Apr 2007
    Posts
    2,010

    Default

    there is already a socket reader and socket writer step. Will they do what you want?

  4. #4
    Join Date
    Mar 2009
    Posts
    137

    Default

    Quote Originally Posted by codek View Post
    there is already a socket reader and socket writer step. Will they do what you want?
    I'm really not sure if those steps can do it... I read or better saw that the PDI can also be used for EAI... The use cases I remember dealing with EAI often use some Input from one socket and write the data to another socket... but the input is used as trigger for the whole transformation...

    I will try to understand those to step you mentioned and tell you if this is a step into the right direction

    Thanks for your answer!

  5. #5
    Join Date
    Mar 2009
    Posts
    137

    Default

    Just for your information... I'm still trying to figure out how I can use those steps for the needs we have... I also read a tdwi article (Data Integration: ETL, EAI, and EII Tools to Create an Integrated Enterprise - http://download.101com.com/tdwi/rese...IRR_Report.pdf).

    My question :
    With the definition of the article of an EAI process (especially: it is event driven) would you say that the PDI can be used for that? Or would you say only partly? or not at all?

    Thanks!

  6. #6
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Endless transformations are quite possible. In PDI 4.0 we will be modifying the logging architecture "a bit" to allow us to better see what's going on during the lifetime of these beasts.

    Obviously "reading from tcp/ip" is a bit too generic. However, since you can write a plugin (it's quite easy actually) AND log data concerning the transformation yourself you can already do what you need to do.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.