Hitachi Vantara Pentaho Community Forums
Results 1 to 15 of 15

Thread: Problem with “File Exists” Job step.

  1. #1

    Default Problem with “File Exists” Job step.

    I am trying to set up my first Kettle job and in doing so I am trying to use a “File Exists” step to check for the existence of a file for processing. If I hard code a file name, the step finds the file and initiate the next transformation step. I have tried several variations to get this step to work with a variable. At this point, I have opted to punt my problem to the people that know something about this stuff.

    My current variable is set as follows (which does not work):
    "MP_JOB_FILE_SEARCH=file:///C:/Projects/CPK/MoneyPenny/extract_bill_utilization_detail_new_*.done$"

    The Job is attached for reference.

    Log Results:
    2007/10/18 10:59:40 - PluginLoader - file:///C:/Documents and Settings/wburrell/.kettle/plugins does not exist.
    2007/10/18 10:59:41 - Spoon - Main window is created.
    2007/10/18 10:59:41 - Spoon - Asking for repository
    2007/10/18 10:59:41 - RepositoriesMeta - Reading repositories XML file: C:\Documents and Settings\wburrell\.kettle\repositories.xml
    2007/10/18 10:59:47 - RepositoriesMeta - Reading repositories XML file: C:\Documents and Settings\wburrell\.kettle\repositories.xml
    2007/10/18 11:00:13 - Spoon - Starting job...
    2007/10/18 11:00:13 - Thread[MoneyPennyLoad (MoneyPennyLoad (Thread-2)),5,main] - Sleeping: 0 minutes
    2007/10/18 11:00:13 - MoneyPennyLoad - Starting entry [File Exists 1]
    2007/10/18 11:00:13 - MoneyPennyLoad - Finished jobentry [File Exists 1] (result=false)
    2007/10/18 11:00:13 - Spoon - Job has ended.

    Thanks.
    Attached Files Attached Files

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    It has to be a single file, not a wild-card.

  3. #3

    Default

    Okay, since the full file name is required, I am trying to follow some of the other examples I found on the forum but I am still having problem getting the file name passed along.

    Check out my job in MoneyPenny.kjb to see how I added another transformation that simply gets the file name and try to set 2 environment variable to be used down stream. Obviously, I am still doing something wrong.
    Attached Files Attached Files

  4. #4

    Question Updated transformations and job

    More recent versions of what I am trying to get to work is attached.
    Attached Files Attached Files

  5. #5
    Join Date
    Mar 2007
    Posts
    216

    Smile

    Hi,

    Quote Originally Posted by wburrell View Post
    More recent versions of what I am trying to get to work is attached.
    If I'm not wrong, the first transformation step you are using is a "Loof for files" step in your "Look for file to process" transformation.

    In this step, I can see ${MP_EXT_DIR} as "File/Directory".

    How PDI is supposed to know what is the content of this variable ?
    I am not a variable addict but I may try to help you.

    a+, =)
    -=Clément=-

  6. #6

    Question

    I have defined some fixed variables in my kettle.properties file. That part is working okay. I am now trying to get the “File Exists” step to work with a variable somehow.

    My current strategy is to:
    1. Have my job run each night and look for a file with a “*.done” extension.
    2. If a “*.done file is found, run the load transformation.
    3. Archived the processed file
    4. Delete the “.done” file.

    These are simple steps that I know have been accomplished routinely before. I may be going about it in the wrong way. If you can suggest a different/better approach, I would appreciate that as well.

    Thanks,

    Warren

  7. #7
    Join Date
    Mar 2007
    Posts
    216

    Smile

    Hi,

    Quote Originally Posted by wburrell View Post
    I have defined some fixed variables in my kettle.properties file. That part is working okay. I am now trying to get the “File Exists” step to work with a variable somehow.
    I wrote a kettle.properties with MP_EXT_DIR in it.
    I deleted your javascript step from the job and the file exists step worked.
    Maybe as soon as variable's type is "valid in the parent job" you do no need to assign them again in the job.
    I could not test the end of our job because it uses a database.

    a+, =)
    -=Clément=-

  8. #8
    Join Date
    May 2006
    Posts
    4,882

    Default

    If you're looking for more than 1 .done file (as I assume you are) you need to use a slighty different way. I would suggest to have a look at the "process all tables" examples in the samples directory.

    You need to execute setVariables once per iteration.

    Regards,
    Sven

  9. #9

    Default

    I actually only need to process 1 file that will be sent on a weekly bases.

    I added place holders for the varibles that I wanted to use in the .properties file. The remaining steps can see the variables but for some reason the "File Exist" works if the extention is a 'txt' but not for '.done'.

  10. #10
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    It works with a ".done" file, I just tested.
    It must me something more subtle.

  11. #11
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Code:
    setVariable("MP_DONE_FILE",filename,"r");
    Change that to:

    Code:
    setVariable("MP_DONE_FILE",filename.getString(),"r");
    Using plain "filename" resorts to "filename.toString()".
    That method by default right-pads to the specified length.

    Obviously, the spaces are hard to see in the log, but they are there ;-)

    HTH,

    Matt

  12. #12

    Default You are the man!

    Thanks for your help.

  13. #13

    Default Now, How does this Zip File object work?

    What is the proper format for the wild card element of the Zip File object.

    It would be awesome if this whole job could be working by end of the day today.....

  14. #14

    Default Never mind.... I figured it out.

    Never mind.... I figured out what the wild card regular expression should have been.

    My completed job is attached for reference.

    Attached Files Attached Files

  15. #15
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Cool!

    Actually, there is a wilcard helper in Eclipse. We should probably steal it and put it in Kettle in some form :-)

    Matt

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.