PDA

View Full Version : Loading weblog data in Hadoop/Hive by using kettle 4.1 transformation



sanjeevsagar
06-29-2011, 12:40 PM
Hello All, I've not found much documentation on this but i'm trying to load the weblog data in Hadoop/Hive by using kettle 4.1

I'm able to create an hadoop/hive database connection but when i use the table output in transformation, it error out on the first line of the input file with the following error




2011/06/29 09:34:15 - hadoopweblogtableout.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : Error inserting/updating row
2011/06/29 09:34:15 - hadoopweblogtableout.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) : Method not supported
2011/06/29 09:34:15 - hadoopweblogtableout.0 - ERROR (version 4.1.0-stable, build 14410 from 2010-11-16 16.43.28 by buildguy) :

Is it possible to load data in Hadoop/Hive hdfs by using kettle transformation?

Meanwhile i can try using the shell script and load the file.

Please let me know.

Highly appreciate it.

jtcornelius
06-29-2011, 01:23 PM
no, the table output step does not currently work for inserting data into Hive via JDBC. At the time we added Hive support, it Hive did not support the necessary INSERT structure needed by the Table Output step. I believe the latest Hive (0.7) now provides this, so we could likely now update our flavor of the driver to support this. I would recommend opening an enhancement for this feature in our Jira system.

-Jake

sanjeevsagar
06-29-2011, 01:53 PM
Thanks Jake !

I also did fine one more post dated: Aug 2010 from Jordan Ganoff :

Support for Hive via the Table Output Step is on our roadmap; however, that functionality will not be included in the GA release. Best estimate I can give you is "soon".

Best Regards,

Jordan Ganoff

So it's not done yet? correct?

Meanwhile i'll send the request in the Jira system.

Thanks Again

jganoff
07-12-2011, 09:28 AM
Correct. Due to limitations in the current state of the Hive JDBC driver this is not possible.

Best,
Jordan