PDA

View Full Version : help with Hadoop Job Executor



kenneththo
11-01-2011, 07:11 PM
hi, i am very new to this environment, but I am also trying out a very simple hadoop job by running the hadoop job executor, I don't see any error but apparently nothing has been done ( i didnt see the job being submitted to the hadoop cluster), the following is the logs, how do i debug it?

2011/11/01 16:07:06 - Spoon - Starting job...
2011/11/01 16:07:06 - hadoop test - Start of job execution
2011/11/01 16:07:06 - hadoop test - exec(0, 0, START.0)
2011/11/01 16:07:06 - START - Starting job entry
2011/11/01 16:07:06 - hadoop test - Starting entry [Hadoop Job Executor]
2011/11/01 16:07:06 - hadoop test - exec(1, 0, Hadoop Job Executor.0)
2011/11/01 16:07:06 - Hadoop Job Executor - Starting job entry
2011/11/01 16:07:06 - Hadoop Job Executor - Using jar path: file:/home/hadoop/hadoop_src/test/wordcount/wordcount.jar
2011/11/01 16:07:06 - Hadoop Job Executor - Running Hadoop Job in Simple Mode
2011/11/01 16:07:06 - hadoop test - Starting entry [Success]
2011/11/01 16:07:06 - hadoop test - exec(2, 0, Success.0)
2011/11/01 16:07:06 - Success - Starting job entry
2011/11/01 16:07:06 - hadoop test - Finished job entry [Success] (result=[true])
2011/11/01 16:07:06 - hadoop test - Finished job entry [Hadoop Job Executor] (result=[true])
2011/11/01 16:07:06 - hadoop test - Job execution finished
2011/11/01 16:07:06 - Spoon - Job has ended.
2011/11/01 16:07:38 - Spoon - Spoon

Jasper
11-03-2011, 05:02 AM
Hi Kenneth,

The thing is when you use the simple version of the job executor the pointers to your hadoop cluster have to be in the jar itself. You have to do some settings for the conf object in java and recompile it.

Its different from when you use the advanced tab, were you can enter the connection details of your Hadoop cluster.

I recognize what you say about the job apparently finishing without problems but in the meantime nothing happens on the Hadoop side. Its frustrating. There is no relevant error catching for this.

Greetz Jasper

kenneththo
11-08-2011, 05:32 PM
yup! that was it. Thank you so much for the tips.




Hi Kenneth,

The thing is when you use the simple version of the job executor the pointers to your hadoop cluster have to be in the jar itself. You have to do some settings for the conf object in java and recompile it.

Its different from when you use the advanced tab, were you can enter the connection details of your Hadoop cluster.

I recognize what you say about the job apparently finishing without problems but in the meantime nothing happens on the Hadoop side. Its frustrating. There is no relevant error catching for this.

Greetz Jasper

probaho
07-21-2012, 01:27 PM
Hi, I am new to pentaho. I am having a similar issue. I have a simple wordcount.jar that works fine outside of spoon. But not working with Hadoop job extractor in simple mode. I have updated the hadoop configuration with host and tasktracker info. Whatelse am I missing here? Can you tell me how you resolved the issue.
My env: windows

2012/07/21 10:16:34 - kettlesamplehadoop - Starting entry [WordCount - Simple]
2012/07/21 10:16:34 - kettlesamplehadoop - exec(2, 0, WordCount - Simple.0)
2012/07/21 10:16:34 - WordCount - Simple - Starting job entry
2012/07/21 10:16:34 - WordCount - Simple - Using jar path: file:/wordCountSample/wordcount.jar
2012/07/21 10:16:34 - WordCount - Simple - Running Hadoop Job in Simple Mode
2012/07/21 10:16:34 - kettlesamplehadoop - Starting entry [Success]
2012/07/21 10:16:34 - kettlesamplehadoop - exec(3, 0, Success.0)
2012/07/21 10:16:34 - Success - Starting job entry
2012/07/21 10:16:34 - kettlesamplehadoop - Finished job entry [Success] (result=[true])
2012/07/21 10:16:34 - kettlesamplehadoop - Finished job entry [WordCount - Simple] (result=[true])
2012/07/21 10:16:34 - kettlesamplehadoop - Finished job entry [Clean Output] (result=[true])
2012/07/21 10:16:34 - kettlesamplehadoop - Job execution finished
2012/07/21 10:16:34 - Spoon - Job has ended.