Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Files consolidation - do you have some recommendations?

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1
    Join Date
    Jun 2016
    Posts
    181

    Default Files consolidation - do you have some recommendations?

    Hi,

    I have a project where I need to transform input file and put it into monthly files directory (each day new file comes).
    Then each day I need to run transformation that takes all files content (original files stay) and produce one consolidated file (month file... this file will contain all day files across the month then new month new one is created). This consolidated information file will be overwritten each day.

    The desired model is that users will have access only to this consolidated file where they can modify some records.
    But the problem is this file should be updated each day; appended new rows from daily files and protect old rows with information modified by users.

    I'm about to abandon this idea and let people work on day files and consolidated file is just to read yet maybe you have some idea how to safely obtain the ideal model?

    Regards,

    G.

  2. #2
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    CSV Output and Excel file writer have "Append" options.

    That doesn't add protection to the existing rows, just adds the new data without changing the existing.

  3. #3
    Join Date
    Jun 2016
    Posts
    181

    Default

    I'm afraid if someone runs transformation twice then same rows will be appended again. However it can work if only some way to check if this is not second run for same file....

  4. #4
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    You could build some sort of check file logic to see if the file has already been processed, and not process it if the filename appears in the list.

  5. #5
    Join Date
    Jun 2016
    Posts
    181

    Default

    I will try this way - thanks

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.