Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Pentaho to CDH 52 hadoop Distro

  1. #1
    Join Date
    Jan 2015
    Posts
    2

    Smile Pentaho to CDH 52 hadoop Distro

    I'm getting the following error while running a MapReduce transformation from Spoon as below. Trying to execute the weblogs parsing ktr downloaded from pentaho wiki (http://wiki.pentaho.com/display/BAD/...Data+into+HDFS)

    Configuration details:
    Spoon Version: 5.3 community Edition installed on Windows 64-bit
    Hadoop: Cloudera 5.2 installed on Linux

    2015/06/05 13:02:21 - Pentaho MapReduce - Starting job entry
    2015/06/05 13:02:21 - Pentaho MapReduce - Using org.apache.hadoop.io.LongWritable for the map output value
    2015/06/05 13:02:21 - Pentaho MapReduce - Using org.apache.hadoop.io.Text for the output key
    2015/06/05 13:02:21 - Pentaho MapReduce - Using org.apache.hadoop.io.LongWritable for the output value
    2015/06/05 13:02:21 - Pentaho MapReduce - Cleaning output path: hdfs://vc2coma1032366n.fmr.com:8020/user/a546885/pdi/weblogs/aggregate_mr
    2015/06/05 13:02:24 - Pentaho MapReduce - Using Kettle installation from /user/a546885/5.3.0.0-213-5.3.0.0-213-cdh52
    2015/06/05 13:02:26 - Pentaho MapReduce - Configuring Pentaho MapReduce job to use Kettle installation from /user/a546885/5.3.0.0-213-5.3.0.0-213-cdh52
    2015/06/05 13:02:27 - Pentaho MapReduce - ERROR (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : Cannot run program "chmod": CreateProcess error=2, The system cannot find the file specified
    2015/06/05 13:02:27 - Pentaho MapReduce - ERROR (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : java.io.IOException: Cannot run program "chmod": CreateProcess error=2, The system cannot find the file specified
    2015/06/05 13:02:27 - Pentaho MapReduce - at java.lang.ProcessBuilder.start(Unknown Source)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.util.Shell.runCommand(Shell.java:444)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.util.Shell.run(Shell.java:414)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:654)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.util.Shell.execCommand(Shell.java:743)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.util.Shell.execCommand(Shell.java:726)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:646)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:434)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:282)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:125)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:348)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
    2015/06/05 13:02:27 - Pentaho MapReduce - at java.security.AccessController.doPrivileged(Native Method)
    2015/06/05 13:02:27 - Pentaho MapReduce - at javax.security.auth.Subject.doAs(Unknown Source)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.pentaho.hadoop.shim.cdh52.HadoopShim.submitJob(HadoopShim.java:83)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.pentaho.hadoop.shim.cdh52.delegating.DelegatingHadoopShim.submitJob(DelegatingHadoopShim.java:112)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.pentaho.di.job.entries.hadooptransjobexecutor.JobEntryHadoopTransJobExecutor.execute(JobEntryHadoopTransJobExecutor.java:866)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.pentaho.di.job.Job.execute(Job.java:716)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.pentaho.di.job.Job.execute(Job.java:547)
    2015/06/05 13:02:27 - Pentaho MapReduce - at org.pentaho.di.job.Job.run(Job.java:424)
    2015/06/05 13:02:27 - Pentaho MapReduce - Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
    2015/06/05 13:02:27 - Pentaho MapReduce - at java.lang.ProcessImpl.create(Native Method)
    2015/06/05 13:02:27 - Pentaho MapReduce - at java.lang.ProcessImpl.<init>(Unknown Source)
    2015/06/05 13:02:27 - Pentaho MapReduce - at java.lang.ProcessImpl.start(Unknown Source)
    2015/06/05 13:02:27 - Pentaho MapReduce - ... 23 more
    2015/06/05 13:02:27 - aggregate_mr - Job execution finished
    2015/06/05 13:02:27 - Spoon - Job has ended.

    Any suggestions to fix this issue?

  2. #2
    Join Date
    Sep 2013
    Posts
    235

    Default

    In this case I guess PMR steps is used as opposite to case when you have your own MR .jar
    When starting MR job first thing PDI do is connect to HDFS and copy himself to HDFS folder. Copy himself means copy all PDI jars and all dependencies into HDFS. So then submitted MR jobs from PDI will be able to use PDI libs - they will be run in HDFS not on local.

    In this case seems to be '/user/a546885/5.3.0.0-213-5.3.0.0-213-cdh52' - then it attempts may attempts to chmod to allow all users to use PDI in their MR jobs.

    'Cannot run program "chmod"'
    My says that user that connected to HDFS have no privileges to chmod - this way PDI can't copy or himself to HDFS or can copy but can't chmod after copy.
    Hope it could help a bit.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.