View Full Version : How to deploy custom plugin to HDFS?

09-23-2013, 12:23 PM
We have built a custom plugin step for Kettle that is working fine locally, but we want to use the plugin in a map-reduce transform. We have already set up the kettle distribution in HDFS under: /opt/pentaho/mapreduce/etc...

I thought we would just copy our whole plugin directory to HDFS:


But I guess not because the map-reduce job fails with:

Unable to load class for step/plugin with id [CamstarStep]. Check if the plugin is available in the plugins subdirectory of the Kettle distribution.

at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:3297)
at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2844)
at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2830)
at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2817)
at org.pentaho.di.trans.TransConfiguration.<init>(TransConfiguration.java:69)
at org.pentaho.di.trans.TransConfiguration.fromXML(TransConfiguration.java:76)
at org.pentaho.hadoop.mapreduce.MRUtil.getTrans(MRUtil.java:68)
at org.pentaho.hadoop.mapreduce.PentahoMapRunnable.createTrans(PentahoMapRunnable.java:195)
... 15 more

How do we deploy a custom step to Hadoop for use in a map-reduce transform?

09-23-2013, 12:30 PM
Did you specify the plugin folder to be included on the nodes in plugin.properties?

# Additional plugins to be copied when Pentaho MapReduce's Kettle Environment does not exist on DFS. This should be a comma-separated
# list of plugin folder names to copy.
# e.g. pmr.kettle.additional.plugins=my-test-plugin,steps/DummyPlugin

So if your "CamstarStep" plugin lives in the "camstar" folder, simply set pmr.kettle.additional.plugins=camstar or steps/camstar for that matter.

(to make sure your plugin is copied over, remove the distributed cache on hdfs /opt/pentaho/... so that the everything is copied over again)