Hi all,

I am having trouble with the "S3 File Output" step in Pentaho 5.3 when running on Linux/CentOS.

I have a 2-step PoC transformation that pushes a bunch of fields into Gz-compressed text file in my S3 bucket. This works fine when running in Windows 7. But when running in Linux I get the following:
Code:
2016/12/22 17:03:34 - TestArea S3 PoC - ERROR (version 5.3-SNAPSHOT, build 1 from 2015-04-23 13.43.15 by mgt) : org.pentaho.di.core.exception.KettleException: 
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1205)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:648)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:716)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:598)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:69)
2016/12/22 17:03:34 - TestArea S3 PoC -     at java.lang.Thread.run(Thread.java:745)
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleMissingPluginsException: 
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2835)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2671)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2623)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2600)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1154)
2016/12/22 17:03:34 - TestArea S3 PoC -     ... 8 more
2016/12/22 17:03:34 - TestArea S3 PoC - ERROR (version 5.3-SNAPSHOT, build 1 from 2015-04-23 13.43.15 by mgt) : org.pentaho.di.core.exception.KettleException: 
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error occurred while launching entry [CREATE GZ FILE ON S3.0]
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:862)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:598)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:69)
2016/12/22 17:03:34 - TestArea S3 PoC -     at java.lang.Thread.run(Thread.java:745)
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleException: 
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1205)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:648)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:716)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC -     ... 5 more
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleMissingPluginsException: 
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2835)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2671)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2623)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2600)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1154)
2016/12/22 17:03:34 - TestArea S3 PoC -     ... 8 more
2016/12/22 17:03:34 - TestArea S3 PoC - ERROR (version 5.3-SNAPSHOT, build 1 from 2015-04-23 13.43.15 by mgt) : org.pentaho.di.core.exception.KettleException: 
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error occurred while launching entry [Get Messages & Property Detail.0]
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error occurred while launching entry [CREATE GZ FILE ON S3.0]
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:862)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:598)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:69)
2016/12/22 17:03:34 - TestArea S3 PoC -     at java.lang.Thread.run(Thread.java:745)
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleException: 
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error occurred while launching entry [CREATE GZ FILE ON S3.0]
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:862)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC -     ... 4 more
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleException: 
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1205)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:648)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:716)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC -     ... 5 more
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleMissingPluginsException: 
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC - 
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2835)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2671)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2623)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2600)
2016/12/22 17:03:34 - TestArea S3 PoC -     at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1154)
2016/12/22 17:03:34 - TestArea S3 PoC -     ... 8 more
org.pentaho.di.core.exception.KettleException:
It sounds like this is related to the following issue:
http://jira.pentaho.com/browse/PDI-13575
(Despite being a different plugin)

In the abovementioned JIRA, a workaround is suggested as:
"As a workaround - the one you already found out - is to call the kitchen/pan from inside the folder the scripts are located."
In the documentation subtask on that page, there is also a sample Shell Script for Spoon named "spoon_5.3.sh".

Looking in this file, the only difference I can see between that and my spoon.sh (which I don't have permission to modify) is that in my spoon.sh I have an extra case statement under "setPentahoEnv" in the "Init BASEDIR" section as follows:
Code:
# **************************************************
# ** Init BASEDIR                                 **
# **************************************************


BASEDIR=`dirname $0`
cd $BASEDIR
DIR=`pwd`
cd -


. "$DIR/set-pentaho-env.sh"


setPentahoEnv


# Spoon really needs to be in the "right" directory
case "$1" in
    # These are non-spoon calls
    -main)
    ;;


    # This is actually spoon
    *)
        cd $DIR
        BASEDIR=.
    ;;
esac

The kitchen.sh script I am using to access this looks like this:
Code:
#!/bin/sh


BASEDIR="`dirname $0`"
cd "$BASEDIR"
DIR="`pwd`"
cd - > /dev/null


if [ "$1" = "-x" ]; then
  set LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$BASEDIR/lib
  export LD_LIBRARY_PATH
  export OPT="-Xruntracer $OPT"
  shift
fi


export IS_KITCHEN="true"


"$DIR/spoon.sh" -main org.pentaho.di.kitchen.Kitchen "$@"
I have interpreted the workaround such that the guts of my shell script (/home/pentaho/testarea/run_s3test.sh) looks like this:
Code:
cd /home/pentaho/testarea
KETTLE_HOME=/home/pentaho/testarea \
PENTAHO_MEM=15G \
/home/pentaho/kettle/kitchen.sh \
  /file:///home/pentaho/testarea/s3_poc.kjb \
  > logs/s3_poc.`date +%Y%m%d`.log 2>&1
Questions:
1) Is this indeed the same issue as assumed above or is it a separate issue.

2) If this is the same issue, then can someone show me how I can modify my run_s3test.sh shell script to make it work?
(If it's the same issue, then my interpretation of the fix is wrong because it ain't workin)

Thanks!!!


Regards,

Stanbridge