Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Need suggestion.

  1. #1

    Default Need suggestion.

    Hi,

    I have set up my hadoop cluster using few ubuntu machines.
    I have installed PDI 4.1 on windows 7.
    I am able to communicate with my hadoop cluster using PDI 4.1.
    I have used 'Hadoop Copy Files' step successfully to copy my local file into HDFS and vice-versa.

    So, I want to know below points:
    1. Is this setup fine?
    2. If not, what problems I can face while using other steps?
    3. I ran few wordcount examples on my Hadoop Clusters which are delievred with my hadoop setup, Is it possible to use the same .jar file to run the same example using PDI 4.1?
    4. Do anybody have any document for other steps as I want to explore these steps?

    Many thanks in advance..

    Thanks,
    Vishwesh

  2. #2
    Join Date
    Aug 2010
    Posts
    87

    Default

    Hi Vishwesh,
    1. Sounds like Hadoop is correctly configured; however, you need to install PDI for Hadoop (PHD) on each node. See Hadoop Node Installation Procedure on page 5 of the Using PDI for Hadoop guide: hadoop_pentaho.pdf
    2. Generally, all steps can work in the transformations used as mappers and reducers. Keep in mind that some steps, such as JMS Consumer/Producer or other long running steps, are not a good idea. If you're using a step that is provided by a plugin or one that requires additional libraries (any step that talks to a database) you need to make sure the required libraries and plugins are installed on all Hadoop nodes.
    3. The Hadoop Job Executor will execute any self-contained jar that can be run on the hadoop command line.
    4. The Getting Started with PDI guide may be a good place to start if you're looking for a high level overview of what you can do with PDI. As stated in response to #2, all steps should generally work with PDI 4.1.0-GA. You can find the getting started guide in the $INSTALL_DIR/data-integration/docs/ folder.


    * Note: Plugins must be placed in $USER_HOME/.kettle/plugins due to a bug in PDI 4.1.0-GA. See PDI-5062 for more information.


    Happy Hadooping,

    Jordan

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.