Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Text File Input Transformation Behavior

  1. #1
    Join Date
    Oct 2008
    Posts
    28

    Unhappy Text File Input Transformation Behavior

    I have built a transformation that I want to run against 6 individual files. Each file has to be run in sequence. File 1 has to processed and insert into table before File 2 etc.

    Now I tried using Text File Input and put all 6 filenames in sequence and run the transformation in one go. But the result seems to be random when I look at the records (i.e. file3 maybe processed ahead of file 1 in some cases). Is this true (maybe it's parallel processed).

    Another way I was thinking is to duplicate the transformation 6 times but that would e a nightmare to maintain.

    My other thoughts is to use a JOB to pass parameters to the transformation but wasn't sure how to do that either.

    Any idea how I can process a transformation 6 times in sequence?

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    You need jobs probably... step entries in a transformation run in parallel. job entries normally run in sequence.

    Regards,
    Sven

  3. #3
    DEinspanjer Guest

    Default

    Regardless of a job or a transformation, the important thing is going to be getting the filenames to the text file input step in the correct order. I believe that if you use the text file input filenames table, it will be based on the order returned by the file system, but for multiple entries in the table, it might be arbitrary, so that probably isn't what you want.
    If the filenames can be sorted lexically, then you could use the Get File Names step to get them and then pass them through a sort step.
    If they are not sortable then the only thing I can think of is this:

    Create six Generate Rows steps. Set each one to output one row, each one to have two fields, a "filename" field with one of the filenames, and an "order" field with a number representing the order desired
    Pass all six of those steps into a sort rows step sorting on the "order" field
    Pass the results of the sort rows step in to the Text File Input step and use the "Accept Filenames From Previous Step" section.

  4. #4
    Join Date
    Oct 2008
    Posts
    28

    Default

    Thanks it works in this case. I have attached a sample.
    Attached Files Attached Files

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.