Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Pentaho PDI between two Hadoop distribution???????

  1. #1
    Join Date
    Dec 2013
    Posts
    10

    Question Pentaho PDI between two Hadoop distribution???????

    Hi all,

    Is it possible to use Pentaho Data Integration (Kettle) between two different Hadoop distributions? Say, your source is some version of Apache or Cloudera and your target is some version of MapR or HartonWorks or something else (am just giving random example). So you are doing your regular Big Data ETLs to extract, polish, cleanse, and load big data from source Hadoop system to target Hadoop system.

    Am interested in knowing, how to configure our Pentaho Kettle (single instance) with these two source and destination Hadoops for Plugin.Properties file, shims etc and run the KTR and Jobs. If available, can you please share the How-To documents?



    ----
    Thanks and Regards,
    Rajeev, Bengaluru, India.
    ____________
    Thanks & Regards,
    Rajeeva Vandakar,
    GrayMatter Software Services Pvt. Ltd.,
    Bengaluru, India.

  2. #2
    Join Date
    Sep 2012
    Posts
    71

    Default

    This is not currently possible using the plugin.properties and Big Data steps, but if your source cluster has WebHDFS enabled, you should be able to use the REST or HTTP steps to get the data into Kettle, then perform ETL, then you can use the shim/Big Data steps to further manipulate the data and get it into your target cluster.

    If the vendors (besides Hortonworks which contributed it) adopt Apache Falcon, this becomes fairly straightforward as it has a REST interface and provides API compatibility between various clusters, so you can use Kettle without actually leveraging the Big Data steps or the Hadoop configuration capability at all.

  3. #3

    Default

    Thank you for the good information..Thanks for sharing...
    inFORM Decisions - Leaders in Document Automation - www.informdecisions.com

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.