Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: CDH5.2 YARN (MR2) Failed without an error message in Kettle 5.2

  1. #1
    Join Date
    Oct 2009
    Posts
    13

    Default CDH5.2 YARN (MR2) Failed without an error message in Kettle 5.2

    I've installed three-node cluster using Cloudera distribution (CDH5.2) in my virtual environment.

    I've also installed Kettle 5.2 in one of the nodes. The connection to HDFS as well as to the YARN cluster seem working fine (By the way, for your reference, see the link for the detail on connectivity to the YARN cluster)

    So, I am now playing around with the the example Pentaho put in http://wiki.pentaho.com/display/BAD/...se+Weblog+Data.
    When I ran the weblog_parse_mr.kjb, it failed without a clear error message. That is, the "Pentaho MapReduce" has a red stop icon after the execution of the job but the logs show completion without any error as below:

    2015/01/08 01:50:54 - Spoon - Starting job...
    2015/01/08 01:50:54 - weblog_parse_mr - Start of job execution
    2015/01/08 01:50:54 - weblog_parse_mr - Starting entry [Pentaho MapReduce]
    2015/01/08 01:50:54 - Pentaho MapReduce - Cleaning output path: hdfs://n1.example.com:8020/user/pdi/weblogs/parse
    2015/01/08 01:50:54 - Pentaho MapReduce - Configuring Pentaho MapReduce job to use Kettle installation from /opt/pentaho/mapreduce/5.2.0.0-5.2.0.0-209-cdh51
    2015/01/08 01:51:05 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0
    2015/01/08 01:51:15 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0
    2015/01/08 01:51:25 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0
    2015/01/08 01:51:35 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0
    2015/01/08 01:51:45 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 21.151321 Reducer Completion: 0.0
    2015/01/08 01:51:55 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 33.333336 Reducer Completion: 0.0
    2015/01/08 01:52:05 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 39.05034 Reducer Completion: 0.0
    2015/01/08 01:52:15 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 66.66667 Reducer Completion: 0.0
    2015/01/08 01:52:25 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 66.66667 Reducer Completion: 0.0
    2015/01/08 01:52:35 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 0.0
    2015/01/08 01:52:45 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 100.0 Reducer Completion: 0.0
    2015/01/08 01:52:56 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0
    2015/01/08 01:53:16 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0
    2015/01/08 01:53:16 - weblog_parse_mr - Finished job entry [Pentaho MapReduce] (result=[false])
    2015/01/08 01:53:16 - weblog_parse_mr - Job execution finished
    2015/01/08 01:53:16 - Spoon - Job has ended.

    Also, looking at the HDFS output folder, the result looked incomplete as below:
    [daniel@n2 data-integration]$ hadoop fs "-ls" /user/pdi/weblogs/parse/
    Found 4 items
    -rw-r--r-- 3 daniel supergroup 0 2015-01-08 01:52 /user/pdi/weblogs/parse/_SUCCESS
    -rw-r--r-- 3 daniel supergroup 27147417 2015-01-08 01:51 /user/pdi/weblogs/parse/part-00000
    -rw-r--r-- 3 daniel supergroup 27132365 2015-01-08 01:52 /user/pdi/weblogs/parse/part-00001
    -rw-r--r-- 3 daniel supergroup 27188268 2015-01-08 01:52 /user/pdi/weblogs/parse/part-00002

    I am a little lost since I don't even know where to look at without any error message in the logs.

    Any idea?

    Thank you,
    Daniel

  2. #2
    Join Date
    Oct 2009
    Posts
    13

    Default

    I was able to see the detail logs from the log file below:
    /var/log/hadoop-yarn/hadoop-cmf-yarn-RESOURCEMANAGER-n1.example.com.log.out

    It turned out that the yarn account didn't have a write privilege in the tmp folder of HDFS. The following command in the yarn host (in my case the first node) fixed the problem.

    hadoop fs -chmod -R 1777 /tmp

    Now, I have a different error, which I need to spend more time to find out the fix. I will update the ticket with the details when I get all resolved.

    Thanks,
    Daniel

  3. #3
    Join Date
    Oct 2009
    Posts
    13

    Default

    I learned that I could see the detail error message through the yarn command as below (the application id can be found from the log file mentioned in my previous comment):

    $ yarn logs -applicationId application_1420841940959_0005
    org.apache.commons.vfs.FileNotFoundException: Could not read from "file:///yarn/nm/usercache/daniel/appcache/application_1420841940959_0005/container_1420841940959_0005_01_000002/job.jar" because it is a not a file.
    at org.apache.commons.vfs.provider.AbstractFileObject.getInputStream(Unknown Source)
    at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(Unknown Source)
    at org.apache.commons.vfs.provider.DefaultURLConnection.getInputStream(Unknown Source)
    at java.net.URL.openStream(URL.java:1037)
    at org.scannotation.archiveiterator.IteratorFactory.create(IteratorFactory.java:34)
    at org.scannotation.AnnotationDB.scanArchives(AnnotationDB.java:291)
    at org.pentaho.di.core.plugins.JarFileCache.getAnnotationDB(JarFileCache.java:58)
    at org.pentaho.di.core.plugins.BasePluginType.findAnnotatedClassFiles(BasePluginType.java:258)
    at org.pentaho.di.core.plugins.BasePluginType.registerPluginJars(BasePluginType.java:555)
    at org.pentaho.di.core.plugins.BasePluginType.searchPlugins(BasePluginType.java:119)
    at org.pentaho.di.core.plugins.PluginRegistry.registerType(PluginRegistry.java:570)
    at org.pentaho.di.core.plugins.PluginRegistry.init(PluginRegistry.java:525)
    at org.pentaho.di.core.KettleClientEnvironment.init(KettleClientEnvironment.java:96)
    at org.pentaho.di.core.KettleEnvironment.init(KettleEnvironment.java:91)
    at org.pentaho.di.core.KettleEnvironment.init(KettleEnvironment.java:69)
    at org.pentaho.hadoop.mapreduce.MRUtil.initKettleEnvironment(MRUtil.java:107)
    at org.pentaho.hadoop.mapreduce.MRUtil.getTrans(MRUtil.java:66)
    at org.pentaho.hadoop.mapreduce.PentahoMapRunnable.createTrans(PentahoMapRunnable.java:221)
    at org.pentaho.hadoop.mapreduce.PentahoMapRunnable.configure(PentahoMapRunnable.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
    Caused by: java.io.FileNotFoundException: /yarn/nm/usercache/daniel/appcache/application_1420841940959_0005/container_1420841940959_0005_01_000002/job.jar (Is a directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at org.apache.commons.vfs.provider.local.LocalFile.doGetInputStream(Unknown Source)
    ... 33 more

    According to a thread I found in Cloudera forum (http://community.cloudera.com/t5/Bat...ghlight/false#), it seemed that there is a problem with my input or output mapper value. But I can't tell what's wrong with it.

    Does anybody has any idea?

    Daniel

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.