Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: "S3 File Output" step does not work with variables for access key or secret key

  1. #1
    Join Date
    Nov 2014
    Posts
    1

    Default "S3 File Output" step does not work with variables for access key or secret key

    I am using the "S3 File Output" step in transformations in Pentaho Data Integration 5.1.0. It works correctly when I specify the AWS access key and secret key using the "Access Key" and "Secret Key" fields in the settings. It also works with I specify the keys in the Filename field using the "s3://access_key:secret_key@s3/bucket/path/file" URL format.

    Now for the problem. I have defined variables for the access and secret keys in my kettle.properties file. When I try to use them in either the two key fields or embedded in the Filename field, it consistently fails with a "Could not determine the type of file" error, which is thrown whenever authentication fails. I have verified that the username and password in the variables are correct--it simply refuses to use the variables correctly. It also fails when I try to use just a variable for the access key. The access key contains just alphanumeric characters, so the problem does not involve incorrect URL encoding of the key. I'd be eager to hear if a workaround is available. Note that this problem is different from bug PDI-10113, in which the key fields did not work at all. The behavior I'm seeing shows that this bug was indeed fixed in this release.

    Here is an example of the log output (with redacted S3 URLs):

    2014/11/15 04:01:25 - S3 File Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Couldn't open file s3://s3/bucket/path/file
    2014/11/15 04:01:25 - S3 File Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : org.pentaho.di.core.exception.KettleException:
    2014/11/15 04:01:25 - S3 File Output.0 - Error opening new file : org.apache.commons.vfs.FileSystemException: Could not determine the type of file "s3://s3/bucket/path".
    2014/11/15 04:01:25 - S3 File Output.0 -
    2014/11/15 04:01:25 - S3 File Output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.openNewFile(TextFileOutput.java:655)
    2014/11/15 04:01:25 - S3 File Output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.init(TextFileOutput.java:754)
    2014/11/15 04:01:25 - S3 File Output.0 - at org.pentaho.di.trans.step.StepInitThread.run(StepInitThread.java:69)
    2014/11/15 04:01:25 - S3 File Output.0 - at java.lang.Thread.run(Thread.java:745)
    2014/11/15 04:01:25 - CDW Phoenix to S3 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!

  2. #2

    Default

    This should be fixed in PDI 5.2, per http://jira.pentaho.com/browse/PDI-12215

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.