Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: SFTP Get with copy previous results to args seems to ignore the results

  1. #1
    Join Date
    Mar 2006
    Posts
    9

    Question SFTP Get with copy previous results to args seems to ignore the results

    I am trying to download some specific files from an SFTP server. I am setting the filenames in a transformation and coping the results to an SFTP get step. However, the SFTP get step ignores my results and downloads the entire directory.

    I have attached screenshots of the tranform and job . The dataflow and logs are below.

    I cant for the life of me figure this out.. What am I dong wrong ? Appreciate the help

    Nikh

    Dataflow
    ==========

    trans_set_download_filenames ( get_filenames -> copy rows to results )
    |
    |
    V
    Get a file with SFTP ( copy previous results to args )


    Log
    ======

    2009/07/17 11:53:13 - job_get_xxxx_files - Starting entry [Get a file with SFTP 1]
    2009/07/17 11:53:13 - Thread[job_get_xxxx_files (job_get_xxxx_files (Thread-85)),5,main] - exec(2, 0, Get a file with SFTP 1.0)
    2009/07/17 11:53:13 - Get a file with SFTP 1 - Start of SFTP job entry
    2009/07/17 11:53:13 - Get a file with SFTP 1 - Getting filename xxxx0717.ou1 from results of previous job entry ...
    2009/07/17 11:53:13 - Get a file with SFTP 1 - Getting filename xxxx0717.de1 from results of previous job entry ...
    2009/07/17 11:53:13 - Get a file with SFTP 1 - Target folder C:\Temp\dwndir exists.
    2009/07/17 11:53:13 - Get a file with SFTP 1 - Opened SFTP connection to server [sftp.downloadserver.com] on port [22] with username [SFTPUSER]
    2009/07/17 11:53:17 - Get a file with SFTP 1 - Changed to directory [/HOME]
    2009/07/17 11:53:18 - Get a file with SFTP 1 - Found 40 files in the remote directory
    2009/07/17 11:53:18 - Get a file with SFTP 1 - Getting file [xxxx0622.de1] to directory [C:\Temp\dwndir]
    2009/07/17 11:53:18 - Get a file with SFTP 1 - Filename xxxx0622.de1 was added to result filenames
    2009/07/17 11:53:18 - Get a file with SFTP 1 - Transferred file [xxxx0622.de1]
    2009/07/17 11:53:18 - Get a file with SFTP 1 - Getting file [xxxx0622.ou1] to directory [C:\Temp\dwndir]
    2009/07/17 11:53:19 - Get a file with SFTP 1 - Filename xxxx0622.ou1 was added to result filenames
    2009/07/17 11:53:19 - Get a file with SFTP 1 - Transferred file [xxxx0622.ou1]
    2009/07/17 11:53:19 - Get a file with SFTP 1 - Getting file [xxxx0623.de1] to directory [C:\Temp\dwndir]
    Attached Images Attached Images    

  2. #2

    Default

    Hi Nickh
    thanks for reporting.
    I cheched and opened a bug here :

    http://jira.pentaho.com/browse/PDI-2541

    Samatar
    Samatar

  3. #3
    Join Date
    Apr 2013
    Posts
    2

    Default

    Name:  write-result-file-name.JPG
Views: 262
Size:  5.2 KBName:  Job.jpg
Views: 260
Size:  5.6 KBName:  mysftp.jpg
Views: 282
Size:  20.9 KBName:  transform-file-name.jpg
Views: 296
Size:  14.4 KB
    Hi,

    I am facing problem in same area. I am using PDI 4.4.0 stable version. Like to know if doing any mistake and/or way I can further dig issue and fix myself.

    Step - 1: Transform file names :: this will read static (for test) file name from data grid; do find replace and set to "result file". in parallel to that it writes same info to text file
    Step - 2: mysftp :: it should read file name that it received from step #1 (Copy Previous Results to Arg is checked). but it does not ... as per debug-log; it didn't received any value!!!
    Step - 3: This is for checking :: Write-result-file-name :: it get's name from previous result and writes same to text file.

    both text file; generated in step 1 and step 3; shows proper file name but mysftp reports not received any thought?

    Thanks for help in advance.


    Thanks,
    YT.

    PS. Log under debug mode -
    [I couldn't find way to attached txt file - log - hence pasted here; sorry for long text ]

    2013/04/03 14:46:05 - Spoon - Starting job...
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - Start of job execution
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - exec(0, 0, START.0)
    2013/04/03 14:46:05 - START - Starting job entry
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - Starting entry [Transform file names]
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - exec(1, 0, Transform file names.0)
    2013/04/03 14:46:05 - Transform file names - Starting job entry
    2013/04/03 14:46:05 - Transform file names - Opening transformation: [file:///C:/....../scripts/pentaho/jobs/rfa/../../transform/rfa//RfaTransformFileNames.ktr]
    2013/04/03 14:46:05 - Transform file names - Loading transformation from XML file [file:///C:/....../scripts/pentaho/jobs/rfa/../../transform/rfa//RfaTransformFileNames.ktr]
    2013/04/03 14:46:05 - Transform file names - Starting transformation...(file=${Internal.Job.Filename.Directory}/../../transform/rfa//RfaTransformFileNames.ktr, name=Transform file names, repinfo=null)
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation is pre-loaded.
    2013/04/03 14:46:05 - RfaTransformFileNames - nr of steps to run : 5 , nr of hops : 4
    2013/04/03 14:46:05 - RfaTransformFileNames - Dispatching started for transformation [RfaTransformFileNames]
    2013/04/03 14:46:05 - RfaTransformFileNames - Nr of arguments detected:10
    2013/04/03 14:46:05 - RfaTransformFileNames - This is not a replay transformation
    2013/04/03 14:46:05 - RfaTransformFileNames - I found 4 different steps to launch.
    2013/04/03 14:46:05 - RfaTransformFileNames - Allocating rowsets...
    2013/04/03 14:46:05 - RfaTransformFileNames - Allocating rowsets for step 0 --> Replace in string
    2013/04/03 14:46:05 - RfaTransformFileNames - prevcopies = 1, nextcopies=1
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation allocated new rowset [Replace in string.0 - Set files in result.0]
    2013/04/03 14:46:05 - RfaTransformFileNames - prevcopies = 1, nextcopies=1
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation allocated new rowset [Replace in string.0 - Text file output.0]
    2013/04/03 14:46:05 - RfaTransformFileNames - Allocated 2 rowsets for step 0 --> Replace in string
    2013/04/03 14:46:05 - RfaTransformFileNames - Allocating rowsets for step 1 --> Set files in result
    2013/04/03 14:46:05 - RfaTransformFileNames - Allocated 2 rowsets for step 1 --> Set files in result
    2013/04/03 14:46:05 - RfaTransformFileNames - Allocating rowsets for step 2 --> Data Grid
    2013/04/03 14:46:05 - RfaTransformFileNames - prevcopies = 1, nextcopies=1
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation allocated new rowset [Data Grid.0 - Replace in string.0]
    2013/04/03 14:46:05 - RfaTransformFileNames - Allocated 3 rowsets for step 2 --> Data Grid
    2013/04/03 14:46:05 - RfaTransformFileNames - Allocating rowsets for step 3 --> Text file output
    2013/04/03 14:46:05 - RfaTransformFileNames - Allocated 3 rowsets for step 3 --> Text file output
    2013/04/03 14:46:05 - RfaTransformFileNames - Allocating Steps & StepData...
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation is about to allocate step [Replace in string] of type [ReplaceString]
    2013/04/03 14:46:05 - RfaTransformFileNames - Step has nrcopies=1
    2013/04/03 14:46:05 - Replace in string.0 - Starting allocation of buffers & new threads...
    2013/04/03 14:46:05 - Replace in string.0 - Step info: nrinput=1 nroutput=2
    2013/04/03 14:46:05 - Replace in string.0 - Got previous step from [Replace in string] #0 --> Data Grid
    2013/04/03 14:46:05 - Replace in string.0 - input rel is 1:1
    2013/04/03 14:46:05 - Replace in string.0 - Found input rowset [Data Grid.0 - Replace in string.0]
    2013/04/03 14:46:05 - Replace in string.0 - output rel. is 1:1
    2013/04/03 14:46:05 - Replace in string.0 - Found output rowset [Replace in string.0 - Set files in result.0]
    2013/04/03 14:46:05 - Replace in string.0 - output rel. is 1:1
    2013/04/03 14:46:05 - Replace in string.0 - Found output rowset [Replace in string.0 - Text file output.0]
    2013/04/03 14:46:05 - Replace in string.0 - Finished dispatching
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation has allocated a new step: [Replace in string].0
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation is about to allocate step [Set files in result] of type [FilesToResult]
    2013/04/03 14:46:05 - RfaTransformFileNames - Step has nrcopies=1
    2013/04/03 14:46:05 - Set files in result.0 - distribution activated
    2013/04/03 14:46:05 - Set files in result.0 - Starting allocation of buffers & new threads...
    2013/04/03 14:46:05 - Set files in result.0 - Step info: nrinput=1 nroutput=0
    2013/04/03 14:46:05 - Set files in result.0 - Got previous step from [Set files in result] #0 --> Replace in string
    2013/04/03 14:46:05 - Set files in result.0 - input rel is 1:1
    2013/04/03 14:46:05 - Set files in result.0 - Found input rowset [Replace in string.0 - Set files in result.0]
    2013/04/03 14:46:05 - Set files in result.0 - Finished dispatching
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation has allocated a new step: [Set files in result].0
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation is about to allocate step [Data Grid] of type [DataGrid]
    2013/04/03 14:46:05 - RfaTransformFileNames - Step has nrcopies=1
    2013/04/03 14:46:05 - Data Grid.0 - distribution activated
    2013/04/03 14:46:05 - Data Grid.0 - Starting allocation of buffers & new threads...
    2013/04/03 14:46:05 - Data Grid.0 - Step info: nrinput=0 nroutput=1
    2013/04/03 14:46:05 - Data Grid.0 - output rel. is 1:1
    2013/04/03 14:46:05 - Data Grid.0 - Found output rowset [Data Grid.0 - Replace in string.0]
    2013/04/03 14:46:05 - Data Grid.0 - Finished dispatching
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation has allocated a new step: [Data Grid].0
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation is about to allocate step [Text file output] of type [TextFileOutput]
    2013/04/03 14:46:05 - RfaTransformFileNames - Step has nrcopies=1
    2013/04/03 14:46:05 - Text file output.0 - distribution activated
    2013/04/03 14:46:05 - Text file output.0 - Starting allocation of buffers & new threads...
    2013/04/03 14:46:05 - Text file output.0 - Step info: nrinput=1 nroutput=0
    2013/04/03 14:46:05 - Text file output.0 - Got previous step from [Text file output] #0 --> Replace in string
    2013/04/03 14:46:05 - Text file output.0 - input rel is 1:1
    2013/04/03 14:46:05 - Text file output.0 - Found input rowset [Replace in string.0 - Text file output.0]
    2013/04/03 14:46:05 - Text file output.0 - Finished dispatching
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation has allocated a new step: [Text file output].0
    2013/04/03 14:46:05 - RfaTransformFileNames - This transformation can be replayed with replay date: 2013/04/03 14:46:05
    2013/04/03 14:46:05 - RfaTransformFileNames - Initialising 4 steps...
    2013/04/03 14:46:05 - Replace in string.0 - Running on slave server #0/1.
    2013/04/03 14:46:05 - Text file output.0 - Running on slave server #0/1.
    2013/04/03 14:46:05 - Data Grid.0 - Running on slave server #0/1.
    2013/04/03 14:46:05 - Set files in result.0 - Running on slave server #0/1.
    2013/04/03 14:46:05 - Text file output.0 - Parent folder exists [file:///C:/Projects/test]!
    2013/04/03 14:46:05 - Text file output.0 - Opening output stream in nocompress mode
    2013/04/03 14:46:05 - Text file output.0 - Opening output stream in default encoding
    2013/04/03 14:46:05 - Text file output.0 - Opened new file with name [C:\Projects\test\Renam-Output.txt]
    2013/04/03 14:46:05 - RfaTransformFileNames - Step [Replace in string.0] initialized flawlessly.
    2013/04/03 14:46:05 - RfaTransformFileNames - Step [Set files in result.0] initialized flawlessly.
    2013/04/03 14:46:05 - RfaTransformFileNames - Step [Data Grid.0] initialized flawlessly.
    2013/04/03 14:46:05 - RfaTransformFileNames - Step [Text file output.0] initialized flawlessly.
    2013/04/03 14:46:05 - RfaTransformFileNames - Transformation has allocated 4 threads and 3 rowsets.
    2013/04/03 14:46:05 - Data Grid.0 - Signaling 'output done' to 1 output rowsets.
    2013/04/03 14:46:05 - Replace in string.0 - Signaling 'output done' to 2 output rowsets.
    2013/04/03 14:46:05 - Set files in result.0 - Added 1 files to the result
    2013/04/03 14:46:05 - Set files in result.0 - Signaling 'output done' to 0 output rowsets.
    2013/04/03 14:46:05 - Text file output.0 - Signaling 'output done' to 0 output rowsets.
    2013/04/03 14:46:05 - Text file output.0 - Closing normal file ...
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - Starting entry [mysftp]
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - exec(2, 0, mysftp.0)
    2013/04/03 14:46:05 - mysftp - Starting job entry
    2013/04/03 14:46:05 - mysftp - Start of SFTP job entry
    2013/04/03 14:46:05 - mysftp - We found 0 row from results of previous job entry!
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - Starting entry [Transformation]
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - exec(3, 0, Transformation.0)
    2013/04/03 14:46:05 - Transformation - Starting job entry
    2013/04/03 14:46:05 - Transformation - Opening transformation: [C:\......\scripts\pentaho\transform\rfa\RfaCheckResultRows]
    2013/04/03 14:46:05 - Transformation - Loading transformation from XML file [C:\......\scripts\pentaho\transform\rfa\RfaCheckResultRows]
    2013/04/03 14:46:05 - Transformation - Starting transformation...(file=C:\......\scripts\pentaho\transform\rfa\RfaCheckResultRows, name=Transformation, repinfo=null)
    2013/04/03 14:46:05 - Transformation - Transformation is pre-loaded.
    2013/04/03 14:46:05 - Transformation - nr of steps to run : 2 , nr of hops : 1
    2013/04/03 14:46:05 - Transformation - Dispatching started for transformation [RfaCheckResultRows]
    2013/04/03 14:46:05 - Transformation - Nr of arguments detected:10
    2013/04/03 14:46:05 - Transformation - This is not a replay transformation
    2013/04/03 14:46:05 - Transformation - I found 2 different steps to launch.
    2013/04/03 14:46:05 - Transformation - Allocating rowsets...
    2013/04/03 14:46:05 - Transformation - Allocating rowsets for step 0 --> Get files from result
    2013/04/03 14:46:05 - Transformation - prevcopies = 1, nextcopies=1
    2013/04/03 14:46:05 - Transformation - Transformation allocated new rowset [Get files from result.0 - Text file output.0]
    2013/04/03 14:46:05 - Transformation - Allocated 1 rowsets for step 0 --> Get files from result
    2013/04/03 14:46:05 - Transformation - Allocating rowsets for step 1 --> Text file output
    2013/04/03 14:46:05 - Transformation - Allocated 1 rowsets for step 1 --> Text file output
    2013/04/03 14:46:05 - Transformation - Allocating Steps & StepData...
    2013/04/03 14:46:05 - Transformation - Transformation is about to allocate step [Get files from result] of type [FilesFromResult]
    2013/04/03 14:46:05 - Transformation - Step has nrcopies=1
    2013/04/03 14:46:05 - Get files from result.0 - distribution activated
    2013/04/03 14:46:05 - Get files from result.0 - Starting allocation of buffers & new threads...
    2013/04/03 14:46:05 - Get files from result.0 - Step info: nrinput=0 nroutput=1
    2013/04/03 14:46:05 - Get files from result.0 - output rel. is 1:1
    2013/04/03 14:46:05 - Get files from result.0 - Found output rowset [Get files from result.0 - Text file output.0]
    2013/04/03 14:46:05 - Get files from result.0 - Finished dispatching
    2013/04/03 14:46:05 - Transformation - Transformation has allocated a new step: [Get files from result].0
    2013/04/03 14:46:05 - Transformation - Transformation is about to allocate step [Text file output] of type [TextFileOutput]
    2013/04/03 14:46:05 - Transformation - Step has nrcopies=1
    2013/04/03 14:46:05 - Text file output.0 - distribution activated
    2013/04/03 14:46:05 - Text file output.0 - Starting allocation of buffers & new threads...
    2013/04/03 14:46:05 - Text file output.0 - Step info: nrinput=1 nroutput=0
    2013/04/03 14:46:05 - Text file output.0 - Got previous step from [Text file output] #0 --> Get files from result
    2013/04/03 14:46:05 - Text file output.0 - input rel is 1:1
    2013/04/03 14:46:05 - Text file output.0 - Found input rowset [Get files from result.0 - Text file output.0]
    2013/04/03 14:46:05 - Text file output.0 - Finished dispatching
    2013/04/03 14:46:05 - Transformation - Transformation has allocated a new step: [Text file output].0
    2013/04/03 14:46:05 - Transformation - This transformation can be replayed with replay date: 2013/04/03 14:46:05
    2013/04/03 14:46:05 - Transformation - Initialising 2 steps...
    2013/04/03 14:46:05 - Text file output.0 - Running on slave server #0/1.
    2013/04/03 14:46:05 - Get files from result.0 - Running on slave server #0/1.
    2013/04/03 14:46:05 - Text file output.0 - Parent folder exists [file:///C:/Projects/test]!
    2013/04/03 14:46:05 - Text file output.0 - Opening output stream in nocompress mode
    2013/04/03 14:46:05 - Text file output.0 - Opening output stream in default encoding
    2013/04/03 14:46:05 - Text file output.0 - Opened new file with name [C:\Projects\test\Result-Row-FileName.txt]
    2013/04/03 14:46:05 - Transformation - Step [Get files from result.0] initialized flawlessly.
    2013/04/03 14:46:05 - Transformation - Step [Text file output.0] initialized flawlessly.
    2013/04/03 14:46:05 - Transformation - Transformation has allocated 2 threads and 1 rowsets.
    2013/04/03 14:46:05 - Text file output.0 - Starting to run...
    2013/04/03 14:46:05 - Get files from result.0 - Starting to run...
    2013/04/03 14:46:05 - Get files from result.0 - Signaling 'output done' to 1 output rowsets.
    2013/04/03 14:46:05 - Get files from result.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
    2013/04/03 14:46:05 - Text file output.0 - Signaling 'output done' to 0 output rowsets.
    2013/04/03 14:46:05 - Text file output.0 - Closing normal file ...
    2013/04/03 14:46:05 - Text file output.0 - Finished processing (I=0, O=1, R=1, W=1, U=0, E=0)
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - Finished job entry [Transformation] (result=[true])
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - Finished job entry [mysftp] (result=[true])
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - Finished job entry [Transform file names] (result=[true])
    2013/04/03 14:46:05 - RfaInitializationLoadFromFTP - Job execution finished
    2013/04/03 14:46:05 - Spoon - Job has ended.

  4. #4
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    "Copy previous results to args" and "Files in Results" are different.

    If you used "Copy Rows to Results" as the OP did, you would have different outcome.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  5. #5
    Join Date
    Apr 2013
    Posts
    2

    Default

    Hi Gutle, Thanks for your reply.

    just fyi... "Get file with secure FTP" property says "copy previous result to arg" that's a reason in my transformation, I sued "set files in result".

    I changed transformation and replaced "set files in result" with "copy rows to result" but still I could not achieve what I wanted to.

    Below are the lines from log (I tried all log level gives same rows for mysft step).

    as we can see it move further, " We found 0 row from results of previous job entry!" is gone now, but
    - it does not download file.
    - as per JobEntrySFTP.java under org.pentaho.di.job.entries.sftp; it is expected to spit out file name sent in argument but it does not

    - previous error gone means it received something arguments but no more data in logs makes me clueless.
    Name:  JobEntrySFTP-java.jpg
Views: 282
Size:  34.6 KB

    ***DEBUG*****
    2013/04/04 13:39:16 - RfaInitializationLoadFromFTP - Starting entry [mysftp]
    2013/04/04 13:39:16 - RfaInitializationLoadFromFTP - exec(2, 0, mysftp.0)
    2013/04/04 13:39:16 - mysftp - Starting job entry
    2013/04/04 13:39:16 - mysftp - Start of SFTP job entry
    2013/04/04 13:39:16 - mysftp - Target folder C:\Projects\test exists.
    2013/04/04 13:39:16 - mysftp - Opened SFTP connection to server [xxx.xxx.xxx.xxx] on port [22] with username [xxxxx]
    2013/04/04 13:39:19 - mysftp - Changed to directory [myftpdir]
    2013/04/04 13:39:21 - mysftp - Found 29 files in the remote directory
    2013/04/04 13:39:21 - RfaInitializationLoadFromFTP - Finished job entry [mysftp] (result=[true])
    2013/04/04 13:39:21 - RfaInitializationLoadFromFTP - Finished job entry [Transform file names] (result=[true])
    2013/04/04 13:39:21 - RfaInitializationLoadFromFTP - Job execution finished


    Thanks,
    YT

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.