Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: HOW: multiple text files output to multiple text files?

  1. #1
    Join Date
    Sep 2005
    Posts
    1,403

    Default HOW: multiple text files output to multiple text files?

    I've got the "Transformation" to work.

    But now I want to apply it to multiple text files and have each loaded text file end up generating another text file with a different extension.

    EXAMPLE:

    xxx_s1.txt will load
    run through a JavaScript
    then try and output
    xxx_s1.mel

    But the output is going to one "mel" file instead of many. I tried to get "Chef" working, but I couldn't figure out how to increment a variable in a file name.

    HOW:

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default RE: HOW: multiple text files output to multiple text files?

    Actually, I'm not sure if this is possible at the moment.
    I have been thinking about a "Loop or Foreach" job entry in Chef, but it needs a bit more thought.

    However, it shouldn't be difficult to script using environment variables.

    Something like :

    for filename in $(ls *_s1.txt)
    do
    pan.sh -file=processit.ktr
    done

    Make sure to set the environment variable filename in pan.sh like this:

    -DFILENAME=${filename}

    Hope this helps.

    Matt

  3. #3
    Join Date
    Feb 2006
    Posts
    3

    Default RE: HOW: multiple text files output to multiple text files?

    I'm running this in CYGWIN. It loops through the files, but gives me a "bad command" error. I have the ktr job in the same folder as the to-be-operated-on text files.

    SCRIPT I'M RUNNING:

    for filename in $(ls sd456789_s*.xyz)
    do
    pan.sh -file=kettle03.ktr
    done


    Is this right? how is the value of "filename" being passed to the spoon job?

  4. #4
    Join Date
    Sep 2005
    Posts
    1,403

    Default HOW: to loop in DOS?

    UPDATE: I couldn't get pan.sh to work on CYGWIN on my 64Bit machine. Dunno why. So I went to DOS.

    I got ONE KTR to work under the DOS command Pan.bat!

    How do I modify the KTR with the command line? say changing the file wildcard?

  5. #5
    Join Date
    Sep 2005
    Posts
    1,403

    Default RE: HOW: multiple text files output to multiple text files?

    The job just keeps writing to the same file over and over. So everything is getting written over a thousand times.

    Is there a setting to prevent that?

  6. #6
    Join Date
    Nov 1999
    Posts
    9,729

    Default RE: HOW: multiple text files output to multiple text files?

    Some quick pointers:
    - use this format under dos (cygwin too): /file:kettle03.ktr
    - add the $filename variable to the java virtual machine so that kettle can read it:
    add option -DFILENAME=$filename
    - specify input file name like %%FILENAME%%
    - specify output file name like %%FILENAME%%.out

    Hope this helps,

    Matt

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.