Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Simple Hadoop Copy Files+Pentaho Trial Pack 5.0.2+Amazon AWS ( Elastic Map Reduce )

  1. #1
    Join Date
    Jan 2014
    Posts
    3

    Question Simple Hadoop Copy Files+Pentaho Trial Pack 5.0.2+Amazon AWS ( Elastic Map Reduce )

    Hello all,

    We have Amazon Web Service with Amazon EMR up and running.

    For building a prototype, I am just trying to use Hadoop Copy Files component to copy one file from local disk to the HDFS.

    So I just added a component Hadoop Copy Files and specified the required Input and Output info , but unfortunately the job is failing during execution.

    Here below I added the full stack trace.
    2014/01/10 10:05:48 - Hadoop Copy Files - ERROR (version 5.0.2, build 1 from 2013-12-04_15-52-25 by buildguy) : Couldn't created parent folder hdfs://machine.compute-1.amazonaws.com:9000/mnt
    2014/01/10 10:05:48 - Hadoop Copy Files - ERROR (version 5.0.2, build 1 from 2013-12-04_15-52-25 by buildguy) : org.apache.commons.vfs.FileSystemException: Could not create folder "hdfs://machine.compute-1.amazonaws.com:9000/".
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.commons.vfs.provider.AbstractFileObject.createFolder(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.commons.vfs.provider.AbstractFileObject.createFolder(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.entries.copyfiles.JobEntryCopyFiles.CreateDestinationFolder(JobEntryCopyFiles.java:667)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.entries.copyfiles.JobEntryCopyFiles.ProcessFileFolder(JobEntryCopyFiles.java:386)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.entries.copyfiles.JobEntryCopyFiles.execute(JobEntryCopyFiles.java:326)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.execute(Job.java:678)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.execute(Job.java:815)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.execute(Job.java:500)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.run(Job.java:407)
    2014/01/10 10:05:48 - Hadoop Copy Files - Caused by: java.io.IOException: Failed on local exception: java.io.IOException: An established connection was aborted by the software in your host machine; Host Details : local host is: "CHEJX05CQ1/2.0.0.2"; destination host is: "machine.compute-1.amazonaws.com":9000;
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client.call(Client.java:1351)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client.call(Client.java:1300)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
    2014/01/10 10:05:48 - Hadoop Copy Files - at com.sun.proxy.$Proxy100.mkdirs(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    2014/01/10 10:05:48 - Hadoop Copy Files - at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at java.lang.reflect.Method.invoke(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    2014/01/10 10:05:48 - Hadoop Copy Files - at com.sun.proxy.$Proxy100.mkdirs(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:467)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2394)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2365)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:817)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:813)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:813)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:806)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1933)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.hdfs.vfs.HDFSFileObject.doCreateFolder(HDFSFileObject.java:81)
    2014/01/10 10:05:48 - Hadoop Copy Files - ... 9 more
    2014/01/10 10:05:48 - Hadoop Copy Files - Caused by: java.io.IOException: An established connection was aborted by the software in your host machine
    2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.SocketDispatcher.read0(Native Method)
    2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.SocketDispatcher.read(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.IOUtil.read(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.FilterInputStream.read(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.FilterInputStream.read(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
    2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.BufferedInputStream.fill(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.BufferedInputStream.read(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.DataInputStream.readInt(Unknown Source)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
    2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
    2014/01/10 10:05:48 - Hadoop Copy Files - ERROR (version 5.0.2, build 1 from 2013-12-04_15-52-25 by buildguy) : Destination folder does not exist!
    2014/01/10 10:05:48 - TestLoadEMR - Finished job entry [Hadoop Copy Files] (result=[false])
    Any Suggestion to make this one work guys.

    Thanks in advance.

    Regards,
    VAP.

  2. #2
    Join Date
    Jan 2014
    Posts
    6

    Default

    You need to enable permission for the hdfs path using hadoop shell command like

    sudo hadoop fs -chmod -R a+rwx /

    Regards,
    Vijay

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.