Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Move files from IN Folder to TOPROCESS folder then ARCHIVE Folder

  1. #1

    Default Move files from IN Folder to TOPROCESS folder then ARCHIVE Folder

    This is my first post so please excuse me if I have not learnt this forum's etiquette yet, or posted it in the wrong place.

    I have an IN Folder that files will arrive in. To these files I wish process them as follows:
    1) Move all files in folder to TOPROCESS folder
    2) Process them and load data into database [This bit I can do ok]
    3) Move all processed files to ARCHIVE folder

    I have a Job that simply does START, Move Files and calls my Transformation to process files and load into database. This move the files and calls the Transformation ok.

    But when the Transformation runs its first step which is Get File Names this does not get any files names despite the files having been moved ok in the Job.

    However if I run the Transformation in Spoon manually then it does get the files names.

    Any suggestions ?

  2. #2
    Join Date
    May 2016
    Posts
    282

    Default

    How are you referencing the folder in the transformation? That's the first thing that comes to my mind... Have you checked the log file for error messages?
    OS: Ubuntu 16.04 64 bits
    Java: Openjdk 1.8.0_131
    Pentaho 6.1 CE

  3. #3

    Default

    Thx for your reply.

    I am working and running things in Spoon. The below is pasted from the Execution Results, Logging tab

    2017/11/17 11:00:53 - Spoon - Starting job...
    2017/11/17 11:00:53 - Tele Move Files - Start of job execution
    2017/11/17 11:00:53 - Tele Move Files - Starting entry [Copy Files]
    2017/11/17 11:00:53 - Copy Files - Starting ...
    2017/11/17 11:00:53 - Copy Files - Processing row source File/folder source : [file:///D:/Telematics/Data/In] ... destination file/folder : [file:///D:/Telematics/Data/ToProcess]... wildcard : [null]
    2017/11/17 11:00:53 - Tele Move Files - Starting entry [Tele_CSV_Load.ktr]
    2017/11/17 11:00:53 - Tele_CSV_Load.ktr - Loading transformation from XML file [file:///D:/Telematics/Transformations/Tele_CSV_Load.ktr]
    2017/11/17 11:00:53 - Tele Move Files - Finished job entry [Tele_CSV_Load.ktr] (result=[true])
    2017/11/17 11:00:53 - Tele Move Files - Finished job entry [Copy Files] (result=[true])
    2017/11/17 11:00:53 - Tele Move Files - Job execution finished
    2017/11/17 11:00:53 - Spoon - Job has ended.

    In the job's Transformatiom step that calls ${Internal.Entry.Current.Directory}/Tele_CSV_Load.ktr is specified a lofile in the Logging tab and pasted below is the contents of the logfile which does not seem much to me being a single line

    2017/11/17 11:00:53 - Tele_CSV_Load.ktr - Loading transformation from XML file [file:///D:/Telematics/Transformations/Tele_CSV_Load.ktr]

    The first step of the transformation is Get File Names specifying File/Directory as D:\Telematics\Data\ToProcess\ with Wilcard (RegExp) as .+(\.csv)$

  4. #4

    Default

    Thx for your reply.

    I am working and running things in Spoon. The below is pasted from the Execution Results, Logging tab

    2017/11/17 11:00:53 - Spoon - Starting job...
    2017/11/17 11:00:53 - Tele Move Files - Start of job execution
    2017/11/17 11:00:53 - Tele Move Files - Starting entry [Copy Files]
    2017/11/17 11:00:53 - Copy Files - Starting ...
    2017/11/17 11:00:53 - Copy Files - Processing row source File/folder source : [file:///D:/Telematics/Data/In] ... destination file/folder : [file:///D:/Telematics/Data/ToProcess]... wildcard : [null]
    2017/11/17 11:00:53 - Tele Move Files - Starting entry [Tele_CSV_Load.ktr]
    2017/11/17 11:00:53 - Tele_CSV_Load.ktr - Loading transformation from XML file [file:///D:/Telematics/Transformations/Tele_CSV_Load.ktr]
    2017/11/17 11:00:53 - Tele Move Files - Finished job entry [Tele_CSV_Load.ktr] (result=[true])
    2017/11/17 11:00:53 - Tele Move Files - Finished job entry [Copy Files] (result=[true])
    2017/11/17 11:00:53 - Tele Move Files - Job execution finished
    2017/11/17 11:00:53 - Spoon - Job has ended.

    In the job's Transformatiom step that calls ${Internal.Entry.Current.Directory}/Tele_CSV_Load.ktr is specified a lofile in the Logging tab and pasted below is the contents of the logfile which does not seem much to me being a single line

    2017/11/17 11:00:53 - Tele_CSV_Load.ktr - Loading transformation from XML file [file:///D:/Telematics/Transformations/Tele_CSV_Load.ktr]

    The first step of the transformation is Get File Names specifying File/Directory as D:\Telematics\Data\ToProcess\ with Wilcard (RegExp) as .+(\.csv)$

  5. #5

    Default

    Ana, Sorry if you thought it took a while for me to reply, but my reply was awaiting moderator approval.

    Thx for your reply.

    I am working and running things in Spoon. The below is pasted from the Execution Results, Logging tab

    2017/11/17 11:00:53 - Spoon - Starting job...
    2017/11/17 11:00:53 - Tele Move Files - Start of job execution
    2017/11/17 11:00:53 - Tele Move Files - Starting entry [Copy Files]
    2017/11/17 11:00:53 - Copy Files - Starting ...
    2017/11/17 11:00:53 - Copy Files - Processing row source File/folder source : [file:///D:/Telematics/Data/In] ... destination file/folder : [file:///D:/Telematics/Data/ToProcess]... wildcard : [null]
    2017/11/17 11:00:53 - Tele Move Files - Starting entry [Tele_CSV_Load.ktr]
    2017/11/17 11:00:53 - Tele_CSV_Load.ktr - Loading transformation from XML file [file:///D:/Telematics/Transformations/Tele_CSV_Load.ktr]
    2017/11/17 11:00:53 - Tele Move Files - Finished job entry [Tele_CSV_Load.ktr] (result=[true])
    2017/11/17 11:00:53 - Tele Move Files - Finished job entry [Copy Files] (result=[true])
    2017/11/17 11:00:53 - Tele Move Files - Job execution finished
    2017/11/17 11:00:53 - Spoon - Job has ended.

    In the job's Transformatiom step that calls ${Internal.Entry.Current.Directory}/Tele_CSV_Load.ktr is specified with a logfile in the Logging tab and pasted below is the contents of the logfile which does not seem much to me being a single line

    2017/11/17 11:00:53 - Tele_CSV_Load.ktr - Loading transformation from XML file [file:///D:/Telematics/Transformations/Tele_CSV_Load.ktr]

    The first step of the transformation is Get File Names specifying File/Directory as D:\Telematics\Data\ToProcess\ with Wilcard (RegExp) as .+(\.csv)$
    Last edited by Bill.Markham; 11-17-2017 at 07:29 AM.

  6. #6

    Default

    I have started from again scratch creating a new Transformation and Job. Building it slowly and carefully with loads of extra diagnostic steps and just adding one step at a time to make sure it works and catch anything dodgy as soon as I had done it and it works !!!!!!!

    Certainly was useful, as you suggested, checking the logs when things went awry.

    Now I just need to carefully remove all my diagnostic steps and tidy it all up.

    Ain't a clue why my first attempt did not work. But there again as a newbie I can't except everything I do to work first time. Thanks for your input.
    Last edited by Bill.Markham; 11-17-2017 at 09:35 AM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.