PDA

View Full Version : Problem connecting PDI4.4 to Hive / Hadoop



Jasper
11-21-2012, 07:57 AM
Hi,

I noticed that I am not the only one that ran into trouble while truying to connect to a Hive instance via JDBC in PDI 4.4.
I have tried multiple setups but just can't get it to work.

I am on PDI 4.4 & CDH3u4 with the standard Cloudera Hive (0.7.1-chd3u4)
Hive thrift server running on the cluster side...


When I use the standard jdbc jar "hive-jdbc-0.7.0-pentaho-1.0.2.jar" from /plugins/pentaho-big-data-plugin/hadoop-configurations/cdh3u4/lib
I get this error message while testing the connection:

############################
Error connecting to database [Hive cHadoop1] : org.pentaho.di.core.exception.KettleDatabaseException:
Error occured while trying to connect to the database

Error connecting to database: (using class org.apache.hadoop.hive.jdbc.HiveDriver)
Error communicating with Hive


org.pentaho.di.core.exception.KettleDatabaseException:
Error occured while trying to connect to the database
#############################

When I tried to use the "hive-jdbc-0.7.0-pentaho-1.0.2.jar" on both end it makes no difference; still this error.



So I tried replacing the "hive-jdbc-0.7.0-pentaho-1.0.2.jar" (on the Pentaho side) with a copy of the "hive-jdbc-0.7.1-cdh3u4.jar" from the cluster

Now I can connect to Hive and browse tables but when I try to access table contents I get the "Method not supported" error message with this stacktrace:

This time I can tell from the Hive logs that the Hive statements is executed; a MR job is being triggered. But when the time comes to stream the results back to Pentaho I get this: (Looks like PDI cannot handle the row results from the Hive query)

#####################################

2012/11/21 12:50:10 - Hive cHadoop1 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : ERROR executing query: org.pentaho.di.core.exception.KettleDatabaseException:
2012/11/21 12:50:10 - Hive cHadoop1 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Error getting row information from database:
2012/11/21 12:50:10 - Hive cHadoop1 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Method not supported
2012/11/21 12:50:10 - Hive cHadoop1 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : ERROR in part: openQuery : get rowinfo
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Unexpected error
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : org.pentaho.di.core.exception.KettleDatabaseException:
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : An error occurred executing SQL in part [openQuery : get rowinfo]:
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : SELECT * FROM gsa_log limit 10
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) :
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Error getting row information from database:
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Method not supported
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) :
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) :
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.core.database.Database.openQuery(Database.java:1863)
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.steps.tableinput.TableInput.doQuery(TableInput.java:233)
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:143)
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.step.RunThread.run(RunThread.java:50)
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at java.lang.Thread.run(Thread.java:662)
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Error getting row information from database:
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Method not supported
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) :
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.core.database.Database.getRowInfo(Database.java:2396)
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.core.database.Database.openQuery(Database.java:1850)
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : ... 4 more
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Caused by: java.sql.SQLException: Method not supported
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.apache.hadoop.hive.jdbc.HiveResultSetMetaData.isSigned(HiveResultSetMetaData.java:177)
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.core.database.Database.getValueFromSQLType(Database.java:2408)
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.core.database.Database.getRowInfo(Database.java:2389)
2012/11/21 12:50:10 - Table input.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : ... 5 more
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Something went wrong while trying to stop the transformation: org.pentaho.di.core.exception.KettleDatabaseException:
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Error cancelling statement
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Method not supported
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : org.pentaho.di.core.exception.KettleDatabaseException:
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Error cancelling statement
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Method not supported
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) :
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.core.database.Database.cancelStatement(Database.java:635)
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.core.database.Database.cancelQuery(Database.java:615)
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.steps.tableinput.TableInput.stopRunning(TableInput.java:303)
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.Trans.stopAll(Trans.java:1563)
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.step.BaseStep.stopAll(BaseStep.java:2525)
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.step.RunThread.run(RunThread.java:74)
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at java.lang.Thread.run(Thread.java:662)
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Caused by: java.sql.SQLException: Method not supported
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.apache.hadoop.hive.jdbc.HiveStatement.cancel(HiveStatement.java:90)
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.core.database.Database.cancelStatement(Database.java:629)
2012/11/21 12:50:10 - Test_Hive - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : ... 6 more
2012/11/21 12:50:10 - Table input.0 - Finished reading query, closing connection.


######################################

lgtleon
12-27-2012, 07:56 AM
Hi!!
I have exactly the same problem. Have you got any solution to solve it?


Thanks!