PDA

View Full Version : fail to run sqoop export/import via kettle



mariax
08-08-2013, 09:38 PM
Hi all

In the big data part, I found two compoents, Sqoop Export/Sqoop Import.
When running sqoop import or export via kettle, I always met some errors.
The error for import:
2013/08/08 16:27:41 - Sqoop Import - ERROR (version 4.4.0-stable, build 17588 from 2012-11-21 16.02.21 by buildguy) : 2013/08/08 16:27:41 - java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.sqoop.mapreduce.RawKeyTextOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1587)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:227)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: Class org.apache.sqoop.mapreduce.RawKeyTextOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1585)
... 8 more

The error for export:
2013/08/08 16:17:35 - Sqoop Export - ERROR (version 4.4.0-stable, build 17588 from 2012-11-21 16.02.21 by buildguy) : 2013/08/08 16:17:35 - java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.sqoop.mapreduce.ExportOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1587)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:227)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: Class org.apache.sqoop.mapreduce.ExportOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1585)
... 8 more

Is there anything that I need to configure?
Thanks.
Maria

mariax
08-12-2013, 04:07 AM
Did anyone meet this issue before?
Is there any solution?

shine.kong
09-22-2013, 03:20 AM
I have the problem too,did someone meet this or had used sqoop import/export successfully?
thanks!

jianjunchu
10-11-2013, 02:05 AM
I have the same problem in windows, and I checked the code and the job.xml file in the Tasktractor.


Finally I found it caused by the classpath in job.xml:


<property><name>mapred.job.classpath.archives</name><value>/home/tekken/hadoop-1.0.3/tmp/mapred/staging/tekken/.staging/job_201310100157_0013/libjars/sqoop-1.4.1-incubating-hadoop-20.jar;/home/tekken/hadoop-1.0.3/tmp/mapred/staging/tekken/.staging/job_201310100157_0013/libjars/mysql-connector-java-3.1.14-bin.jar</value></property>


the seperator in classpath is ";" not ":"


I am not sure whether it is a bug.
But I fixed the code, Added the following statement in the method executeJobEnry of AbstractSqoopJobEntry class


System.setProperty("path.separator",":");


I attached two classes as a patch of this problem. (tested on kettle 4.4 windows)


you can download the two classes, and replace the same class files in the pentaho-big-data-plugin-TRUNK-SNAPSHOT.jar


I think It will works!

jianjunchu
10-11-2013, 02:22 AM
patch file