Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Configuring Dynamic Filenames ?

  1. #1
    Join Date
    Apr 2009
    Posts
    28

    Question Configuring Dynamic Filenames ?

    Hi all,
    i am using kettle 3.1 version , is there any possibility of defining dynamic filenames in the ETL process..

  2. #2
    Join Date
    Apr 2009
    Posts
    28

    Default

    Lets say:

    I am maintaing the historical data in ETL_KETTLE_JOB_LOG file where it maintains the counts for each job.

    Say

    For Campaign present count in ETL_KETTLE_JOB_LOG file is 10 , then

    for next run of ETL i need a way such that i have to extract the data from Campaign( from source) to a flat file named Campaign_11.txt

    and later ill get the data from Campaign_11.txt to wsowner.campaign(staging).

    Reason: to get data from oltp--> flatfile --->staging instead of
    oltp-->staging

    bcz in 2nd process oltp connection is open until the process get loaded in staging.. while with flatfile it is minimized...

    so please help me out.. how can name the flat file dynamically..(for next run flat file name should be Campaing_12.txt")

    Note: i am using "Serialize to file" and "UnSerialize from file" for gettting data from oltp and sending flatfile to staging.



    Regards
    Narendar
    Last edited by viswanathamn; 08-10-2009 at 04:58 AM.

  3. #3

    Default

    Quote Originally Posted by viswanathamn View Post
    Lets say:

    I am maintaing the historical data in ETL_KETTLE_JOB_LOG file where it maintains the counts for each job.

    Say

    For Campaign present count in ETL_KETTLE_JOB_LOG file is 10 , then

    for next run of ETL i need a way such that i have to extract the data from Campaign( from source) to a flat file named Campaign_11.txt

    and later ill get the data from Campaign_11.txt to wsowner.campaign(staging).

    Reason: to get data from oltp--> flatfile --->staging instead of
    oltp-->staging

    bcz in 2nd process oltp connection is open until the process get loaded in staging.. while with flatfile it is minimized...

    so please help me out.. how can name the flat file dynamically..(for next run flat file name should be Campaing_12.txt")

    Note: i am using "Serialize to file" and "UnSerialize from file" for gettting data from oltp and sending flatfile to staging.



    Regards
    Narendar
    What is the reason to get the data into flat file ?

  4. #4
    Join Date
    Apr 2009
    Posts
    28

    Default

    Already i explained in my requirement..

    i need to create filenames dynamically by getting the latest count of the job from the log file i.e. ETL_KETTLE_JOB_LOG and storing a filename as

    Campaign_13.txt (where campaign is the jobname and 12 is the latest count for campaing so 12+1=13 which is for new file name ie campaign_13.txt)

    Please let me know how to do this in kettle.

    Regards
    Narendar.

  5. #5
    Join Date
    Apr 2009
    Posts
    28

    Default

    How can i do dis.. is it possible to do in kettle first of all!!!!

  6. #6
    Join Date
    Feb 2009
    Posts
    296

    Default

    Yes you can.

    The magic is hidden behind the field entitled "Accept file name from field". You'll need to use the Text File Input, maybe a Sort Rows and certainly a Group By step.

    I'm not going to explain the specifics - just go ahead and click around like the rest of us.
    Fabian,
    doing ETL with his hands bound on his back

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.