Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Importing last modified file containing specific text

  1. #1
    Join Date
    Nov 2013
    Posts
    6

    Question Importing last modified file containing specific text

    Hi,

    I am working on a data transformation task that involves fetching data from an input CSV file then converting the data into specific a format and finally exporting the CSV file. I have created the transformation job and I am able to fetch data from a specific CSV file and transform the data.

    However, now the real world situation is that I would be getting the input CSV file every 30 mins, so how can I program this process to fetch only the last modified file that contains specific string.

    For instance: I have following 4 files in a folder:
    Site_A_Data_201311190830.csv
    Site_A_Data_201311190900.csv
    Site_A_Data_201311190930.csv
    Site_A_Data_201311191000.csv

    So, I want the Step "CSV File Input" to open last modified file from above four files that contains string "Site".

    The reason for having the constraint of opening file containing string "Site" is because the output CSV file will be saved in the same directory as well. Also, I want to add a "Send Email" step that would send email with an attachment of last modified output file containing specific string.

    I would really appreciate if someone can please have a look at the attached screen shot and assist me.

    Thanks.

    Name:  Kettle Data Transformation.jpg
Views: 342
Size:  9.7 KB

  2. #2
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Instead of going:
    CSV Input -> MJSV -> Text File Output

    Try going:
    Get Filenames (Regular Expression ".*site.*")
    Sort Rows (lastmodifiedtime Descending)
    Group By (No Group, Statistic on filename , first non-null value)
    CSV Input (get field from "The filename field (data from previous steps)")
    [Processing Step(s)]
    Text File Output
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  3. #3
    Join Date
    Nov 2013
    Posts
    6

    Default

    Thanks for your response mate. I will give it a shot.

    FYI: I wasn't looking for someone to do the work for me. I was just stuck with this issue and needed some assistance with it (I guess that's the idea behind this support forum). Pardon me if my post sent a wrong message to you. Anyways appreciate your assistance.

    Cheers.

  4. #4
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    No offense taken - it's part of my sig, as there are a lot of people on this board who do expect the forum members to actually do the work for them.

    If you have wording that would make it clearer that I'm willing to provide assistance but not do the work for another person on the board, I'd be happy to hear it.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  5. #5
    Join Date
    Nov 2013
    Posts
    6

    Default

    Hi Mate,

    Sorry for the late reply, thanks for the solution. It works for me. Cheers.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.