Hitachi Vantara Pentaho Community Forums
Results 1 to 17 of 17

Thread: VFS question

  1. #1
    Join Date
    Sep 2008
    Posts
    25

    Default VFS question

    I am trying to get a list of files from a remote ftp site. When I try this I get the following error.

    2008/10/31 14:28:39 - FileInputList - ERROR (version 3.1.0, build 826 from 2008/09/30 11:30:46) : java.io.IOException: Unable to get VFS File object for filename 'ftp://xxx:xxx@ftp2.omniture.com:21' : org.apache.commons.vfs.FileSystemException: Could not connect to FTP server on "ftp2.omniture.com".

    I am able to connect and download file with the get ftp files using the same username and password on that site. Do I need to configure something first in order to use vfs?

    Thanks,
    Pete

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    No configuration is needed.
    You do need to specify a file on the server though.

    Code:
    ftp://user:pass@ftp.example.com/path/to/file.txt
    Something along those lines.
    The port is optional:

    http://commons.apache.org/vfs/filesystems.html

  3. #3
    Join Date
    Sep 2008
    Posts
    25

    Default

    I was able to get it work with FTP but I am not able to get it to work with SFTP. The connection does work with get sftp files.

    I used the following code:
    sftp://xxx:xxx@ftp.sharp.com/u02/ftp/add_to_bags.log

    I got the following error:
    2008/10/31 15:11:15 - FileInputList - ERROR (version 3.1.0, build 826 from 2008/09/30 11:30:46) : java.io.IOException: Unable to get VFS File object for filename 'sftp://xxx:xxx@ftp.sharp.com' : org.apache.commons.vfs.FileSystemException: Could not connect to SFTP server at "sftp://xxx:xxx@ftp.sharp.com/"

    Any help is greatly appreciated.

  4. #4

    Default VFS SFTP problem in Kettle v3.2

    Peter,

    I ran into a similar problem and similar error using GeoKettle based on Kettle v3.2.0: After running Spoon in a debug session in Eclipse, I found that the underlying (swallowed) exception mentioned not knowing the host's key:
    Code:
    throw new JSchException("UnknownHostKey: "+host+". "+key_type+" key fingerprint is "+key_fprint);
    I found out that this was because the configuration of SFTP is hardcoded to be 'StrictHostKeyChecking=true' which means that it will not accept hosts that are new to the underlying SSH client (Jsch)...

    One (of the 2) Pentaho Wiki pages on VFS that I found, suggests setting a parameter like this:
    Code:
    vfs.sftp.StrictHostKeyChecking.myservername=true
    However, from what I can see after examining the source code, there's no way to specify this parameter as the VFS Commons code takes it from a 'filesystemOptions' variable that is (now inside Kettle code) hardcoded to be null (because the related 'basefile' it provides to the VFS code is null).

    In other words: When looking to find why my attempts to specify such a parameter had no effect, I got into the impression that the (v3.2.0 of the) code doesn't take any parameters into account.

    This thinking brought me to the JIRA issue 2791 with a rather vague description that should have been fixed by now (as of v3.2.3)... I am now trying to get some recent Kettle (only, as GeoKettle is still based on v3.2.0) code to try and reproduce the testcase. I think the JIRA issue is related, I will now try to find out whether the patch solved our problem...

    Frederik

    BTW: The 'Get SFTP' Job you mentioned, is no good for me as it only works with password authentication, besides the obvious problem of needing to make a local copy of the files...

  5. #5
    Join Date
    Mar 2010
    Posts
    159

    Default

    Only a workaround but hopefully establishing a connection once "manually" using the library should get past the unknown host exception.

  6. #6
    Join Date
    Nov 1999
    Posts
    9,729

  7. #7

    Default workaround NOK

    Quote Originally Posted by jbeal View Post
    Only a workaround but hopefully establishing a connection once "manually" using the library should get past the unknown host exception.
    That workaround didn't work for me:
    I ran Spoon inside an Eclipse debug session and initialised the variabel 'strictHostKeyChecking' so that this conditional code got executed:
    Code:
    if (strictHostKeyChecking != null)
                {
                    config.setProperty("StrictHostKeyChecking", strictHostKeyChecking);
                }
    That run in debug mode it did work, but the run I did afterwards didn't work anymore: Once again "UnknownHostKey: versioning. RSA key fingerprint is d0:62:25:bc:04:ac:93:d1:85:ee:0b:b5:16:11:db:10"...

    Maybe the library isn't persisting the host keys?!

  8. #8

    Default

    Quote Originally Posted by MattCasters View Post
    Is this as a result of the changes made for JIRA-2791, iow. is upgrading to v3.2.3 already enough, or do you know of other changes regarding the support for VFS that were only committed in v4?

    BTW: Do you have any news on the plans regarding v4 by the GeoKettle team?

  9. #9

    Default Will VFS be supported in the future?

    As I use PuTTY as the other SSH client, no `known_hosts` file gets created neither to test the workaround suggested by jbeal.

    I could try and install a third SSH client that does create it (as I see in the VFS source that it is trying to use such a file), but anyway I think it doesn't make any sense to try and have a `known_hosts` because of the known issues with such a file and that it would still only work with password authentication...

    What I really needed is the bugfix to enable VFS parameters (so that I can instruct VFS to accept any key), something that was apparently (I checked this yesterday) done in a candidate release of v4 (build 2572).

    Still 2 questions remain though:

    1. Can I just upgrade the v3.2.0 installation I want to use (because of the GIS extensions) so that it supports reading (big) remote files w/o the need to persist them locally? Or should I go for a port af the GIS features to v4?
    2. Is it worth considering this: How come that the only 2 Pentaho Wiki pages on VFS remain incomplete (this one doesn't talk about VFS parameters and the one I (and later on Matt) mention(s), references an example that is still nowhere to be found). Add to that the Apache VFS community that isn't very active lately: Is this a 'dying' feature, a try-out that has been abandonned?

  10. #10
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Wow, talk about shooting in all directions.

    Here are the answers:
    - For information about the GeoKettle project, contact our friends at GeoKettle / Spacialitics.
    - VFS is here to stay. The software evolves slowly like most Apache software. However, it *is* open source and we have made changes to it in the past. If needed we'll fork it in the future. (I doubt that's going to be needed)

    The SFTP VFS features described in PDI-2791 are indeed available in 3.2.3 and 4.0

    Matt

  11. #11
    Join Date
    Jan 2012
    Posts
    3

    Default Pentaho VFS Issue: Help Needed

    Hi,

    I am able to send pentaho generated XLS files to FTP Server by using the following URL and parameters;
    vfs-ftp://{user}:{password}@{ftp_host}/{directory}/{reportname}.{format}

    Now, i wanted to send it to SFTP instead of FTP. Keys are authenticated on source as well as on Dest and no password is needed. Could anyone help me know how to make the SFTP work. Do i need to add keys anywhere in the Xaction as input parameter? If yes, where and how? Please help. It's a crisis situation for me and m pretty new to Pentaho.

  12. #12
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Did you even read anything in this old thread?

  13. #13
    Join Date
    Jan 2012
    Posts
    3

    Default

    I did check the forum.

    I tried using the following :

    vfs-sftp://{user}:{password}@{ftp_host}/{directory}/{reportname}.{format}

    It didn’t work. Then again, I had a look here http://wiki.pentaho.com/display/COM/...ing+Kettle+VFS and I could know that I should use

    vfs.sftp.StrictHostKeyChecking but its format is different than the above url and I tried its different versions but it didn’t work out.

    Could you help me with the exact url containing all parameters with something like this vfs.sftp.StrictHostKeyChecking://{user}:{password}@{ftp_host}/{directory}/{reportname}.{format}

    Also, when I say that keys are installed on both the Source and destination, I mean on Shell. I believe the mechanism on Java is different.

    Any help shall be greatly appreciated.I did check the forum.
    Last edited by mihirjha27; 01-13-2012 at 08:07 AM. Reason: double post

  14. #14
    Join Date
    Nov 1999
    Posts
    9,729

    Angry

    You're just randomly throwing stuff out there, aren't you?
    I can't decide whether or not you're trying to make fun or if you're serious.
    Whatever may be: please don't ever email me directly anymore.


    sftp://{user}:{password}@{ftp_host}/{directory}/{reportname}.{format}

    example

    sftp://user@pwd4user@ftp.test.com/foo/report.csv


    vfs.sftp.StrictHostKeyChecking --> these are environment variables for crying out loud, not part of the URL.

  15. #15
    Join Date
    Jan 2012
    Posts
    3

    Default

    Matt, i am sorry if it sounds silly here but i am pretty new and naive to it and am not firing things in air.

    I have used the url as suggested by you and now i understand that i need to change the environment variables for the xaction to follow it. I have two questions now if you could help me with:

    1. What is the filename where i can add these variables? Is it the server.xml file in tomcat/conf given that i am trying to send the file to a remote location using private ssh keys.
    2. Pentaho filesystems specific note tells me to add the host in known hosts file in (~/.ssh/known_hosts). It's added for me. But where do i mention the path for Identity and which folder should i put the ssh key and what should be the extension of that file consisting of ssh keys.

    Any help shall be greatly appreacited. Am totally directionless and confused with this. Thanks in advance !
    Last edited by mihirjha27; 01-15-2012 at 08:03 AM. Reason: typo

  16. #16

    Question

    Dear fellow, i have a question about VFS & Parameters:
    I'm working with vfs to sftp sftp://{user}:{password}@{ftp_host}/{directory}/{reportname}.{format} without problem and i use in Transformation Setting - Parameters Tab the sftp.StrictHostKeyChecking and vfs.sftp.proxyHost configurated. But in my office needs to pass the transformation setting parameters in a variable... and I try to this but the table setting not has parameters or variable. Can unferstand me?
    Add my transformation

    VFS TEST Get_Filename_FTP.ktr

    For more example attach image explain i like to do:
    Name:  VFS PARAMETERS.jpg
Views: 99
Size:  26.3 KB

    I hope answ soon... Regards!
    Delia

  17. #17

    Default

    Somebody can say something about this? Pls, i hope you answer..
    Thanks.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.