PDA

View Full Version : How to deploy custom plugin to HDFS?



better
09-23-2013, 12:23 PM
We have built a custom plugin step for Kettle that is working fine locally, but we want to use the plugin in a map-reduce transform. We have already set up the kettle distribution in HDFS under: /opt/pentaho/mapreduce/etc...

I thought we would just copy our whole plugin directory to HDFS:

/opt/pentaho/mapreduce/4.4.2-GA-1.3.4-cdh42/plugins/CamstarStep

But I guess not because the map-reduce job fails with:


Unable to load class for step/plugin with id [CamstarStep]. Check if the plugin is available in the plugins subdirectory of the Kettle distribution.

at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:3297)
at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2844)
at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2830)
at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2817)
at org.pentaho.di.trans.TransConfiguration.<init>(TransConfiguration.java:69)
at org.pentaho.di.trans.TransConfiguration.fromXML(TransConfiguration.java:76)
at org.pentaho.hadoop.mapreduce.MRUtil.getTrans(MRUtil.java:68)
at org.pentaho.hadoop.mapreduce.PentahoMapRunnable.createTrans(PentahoMapRunnable.java:195)
... 15 more

How do we deploy a custom step to Hadoop for use in a map-reduce transform?

MattCasters
09-23-2013, 12:30 PM
Did you specify the plugin folder to be included on the nodes in plugin.properties?


# Additional plugins to be copied when Pentaho MapReduce's Kettle Environment does not exist on DFS. This should be a comma-separated
# list of plugin folder names to copy.
# e.g. pmr.kettle.additional.plugins=my-test-plugin,steps/DummyPlugin
pmr.kettle.additional.plugins=

So if your "CamstarStep" plugin lives in the "camstar" folder, simply set pmr.kettle.additional.plugins=camstar or steps/camstar for that matter.

(to make sure your plugin is copied over, remove the distributed cache on hdfs /opt/pentaho/... so that the everything is copied over again)