Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: SFTP Subfolder Suport

  1. #1

    Default SFTP Subfolder Suport

    I have a use case where I have an SFTP server that a client places files into subfolders. I do not know the name or how many subfolders may be created. Is there a way using Get Files from SFTP step that subfolders can be traversed? If not what is the alternative for walking a directory tree on an SFTP server that can be used to processed the files that are placed there

  2. #2
    Join Date
    Aug 2011
    Posts
    236

    Default

    Hi Jerms,

    Not sure what version of PDI you are using, but in 5.0.1, there is a 'Read subdirectories' component for transformations. I accepts a hard coded main directory or as a parameter.

    You could send the output to 'Copy to results'. Then you can loop around the results to read all files within the directory in the next transformation'

    Job1 -> Trans1 - Read all subdirectories -> Trans2 (Check 'Execute for each input row' in Advanced settings) Read all files in subdirectories.

    HTH
    PDI 8.0.0
    MySQL - 5.6.27
    Redshift - 1.0.1485
    PostgreSQL 8.0.2
    OS - Ubuntu 10.04.2

  3. #3

    Default

    How would I use the SFTP connection then to pass the directory that is on the SFTP server to the substransformation and the Get subfoldernames step? It appears that it is a local directory that is required to be configured.

  4. #4
    Join Date
    Aug 2011
    Posts
    236

    Default

    Hi Jerms,

    Try using the wildcard ('*') to see if it picks up folders as well as files.
    PDI 8.0.0
    MySQL - 5.6.27
    Redshift - 1.0.1485
    PostgreSQL 8.0.2
    OS - Ubuntu 10.04.2

  5. #5
    Join Date
    Apr 2008
    Posts
    4,686

    Default

    Quote Originally Posted by jerms.wilson@gmail.com View Post
    How would I use the SFTP connection then to pass the directory that is on the SFTP server to the substransformation and the Get subfoldernames step? It appears that it is a local directory that is required to be configured.
    VFS is your friend.
    You can build a URI like SFTP://user@host/path/that/you/want/to/look/in and pass that as a parameter to the transformation, and then use that parameter in the Get Subfolder Names step.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.