PDA

View Full Version : Unable to execute a map reduce job - ClassNotFoundException: org.owasp.esapi.ESAPI



rendel
06-19-2013, 07:34 AM
Hi,

I am trying to execute the hadoop examples wordcount and weblogs-parser with the latest PDI distribution (4.4.1-GA). When I am executing the mapreduce job, the following error occurs at the end of the job.



2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : Hadoop Output - Unexpected error
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : Hadoop Output - java.lang.NoClassDefFoundError: org/owasp/esapi/ESAPI
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.pentaho.di.core.xml.XMLHandler.addTagValue(XMLHandler.java:715)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.pentaho.di.core.xml.XMLHandler.addTagValue(XMLHandler.java:772)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.pentaho.di.core.row.ValueMeta.getMetaXML(ValueMeta.java:2491)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.pentaho.di.core.row.RowMeta.getMetaXML(RowMeta.java:898)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.pentaho.hadoop.mapreduce.OutputCollectorRowListener.rowWrittenEvent(OutputCollectorRowListener.java:101)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.pentaho.di.trans.step.BaseStep.putRow(BaseStep.java:1133)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.pentaho.di.trans.steps.hadoopexit.HadoopExit.processRow(HadoopExit.java:69)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.pentaho.di.trans.step.RunThread.run(RunThread.java:50)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.lang.Thread.run(Thread.java:662)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : Caused by: java.lang.ClassNotFoundException: org.owasp.esapi.ESAPI
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.security.AccessController.doPrivileged(Native Method)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
2013/06/19 12:07:54 - Pentaho MapReduce (Web Log Parsing) - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : ... 9 more


Do you have any idea of the cause to this problem ?


Thanks

mattb_pdi
06-19-2013, 08:43 AM
This is a known issue that will be fixed in 4.4.2 and 5.0:

http://jira.pentaho.com/browse/PDI-9867

The case has a workaround listed:

Workaround: You can find the missing jar in data-integration/libext/esapi-2.0.1.jar, place that in the data-integration/plugins/pentaho-big-data-plugin/pentaho-mapreduce-libraries.zip and the problem should be resolved.

rendel
06-19-2013, 11:05 AM
Thanks,
that works.

ieandd
07-14-2013, 02:46 PM
Hi,

I have same problem. I copied esapi-2.0.1.jar and ESAPI.properties nearly every folder in pentaho-mapreduce-libraries.zip:

Copied /
/org/
/org/owasp/
/org/owasp/esapi
/org/owasp/esapi/ESAPI

but i have same error java.lang.NoClassDefFoundError: org/owasp/esapi/ESAPI. Can you help me ?

mattb_pdi
07-14-2013, 06:56 PM
If you have already tried to run Pentaho MapReduce on your cluster, then it won't be reinstalled after you insert the ESAPI JAR and properties file. To force a reinstall, delete the /opt/pentaho/mapreduce folder on the HDFS of your cluster, then retry with Spoon and your augmented pentaho-mapreduce-libraries.zip, and please let us know if it doesn't work.

Regards,
Matt

ieandd
07-15-2013, 04:22 PM
Dear Matt,

Thank you for your reply. It worked. But I have another classnotFound error:

Im using cdh4 my Plugin.properties :11382

My log:



2013/07/15 23:07:33 - Spoon - Starting job...
2013/07/15 23:07:33 - ykb-main - Start of job execution
2013/07/15 23:07:33 - ykb-main - Starting entry [Pentaho MapReduce]
2013/07/15 23:07:33 - Pentaho MapReduce - Cleaning output path: hdfs://n1.example.com:8020/user/oyt/sonuc
2013/07/15 23:07:33 - Pentaho MapReduce - Configuring Pentaho MapReduce job to use Kettle installation from /opt/pentaho/mapreduce/4.4.1-1.3.1-cdh4
2013/07/15 23:07:33 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0
2013/07/15 23:07:38 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0
2013/07/15 23:07:43 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0
2013/07/15 23:07:48 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : [FAILED] -- Task: attempt_201307151358_0011_m_000000_0 Attempt: attempt_201307151358_0011_m_000000_0 Event: 1
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : Error: java.lang.ClassNotFoundException: org.pentaho.di.core.exception.KettleException
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.security.AccessController.doPrivileged(Native Method)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.lang.Class.forName0(Native Method)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at java.lang.Class.forName(Class.java:247)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1526)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1491)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1585)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1611)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.apache.hadoop.mapred.JobConf.getMapRunnerClass(JobConf.java:1010)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:414)
2013/07/15 23:07:48 - Pentaho MapReduce - ERROR (version 4.4.1-GA, build 18329 from 2013-04-17 12.31.49 by buildguy) : at org.apache.hadoop.

Thanks for your help

Regards,

Oytun

ieandd
07-16-2013, 07:57 AM
Hi Matt,

I realized that my conf is wrong. I have cdh4.2.0 cluster and my big-data-plugin is outdated.

I will follow your steps at second post of this thread : http://forums.pentaho.com/showthread.php?144919-CDH-Hive2-connector

We can close this thread.

Thank you,

Regards,