Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Passing Params from Text File

  1. #1

    Question Passing Params from Text File

    I want to define parameters for Kettle jobs in a text file (params.txt) and pass it on the the jobs.

    My idea is

    Step 1

    - Create text file with parameter data (name=value)
    - Use 'File Input' step to load the file
    - Get the fields values from 'Fields' in File Input

    Step 2

    - Use Java Script to pull out value for each field
    - Use the value thus extracted as parameter value for the next step

    The reasoning behind this is to - define the parameters in a text file so that it's easier to configure it from the text file instead of going to individual Kettle jobs to do it.

    A way to improve portability and maintainabiity of Kettle jobs.

    I would really appreciate your help as we have been toying with this idea for a while...

    It may be bothersome to note - we are still using Kettle 2.3.0 :-)

    Thanks!!!

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    1) you can't use variables in the same transformation where you defined them. You can, but they won't work right.

    2) Until 3.0 (not 2.3, but 3.0) variables will have problems when used in combination with more than 1 subjob in a hierachy... a job executing a transformation will work, a job running a subjob running a transformation may have problems.

    Why not put the variables in the kettle.properties file?

    Regards,
    Sven

  3. #3
    Join Date
    May 2007
    Posts
    13

    Default

    I implemented this scheme, more or less, in 2.5.1. I don't know if it is compatible with earlier releases.
    It depends on a command line argument to indicate where the properties file resides. Note that it
    ignores comment lines.

    My idea was to drive the processing from an argument which specifies where the properties file resides.
    I am running on different machines against different database servers and/or database instances.
    There are multiple projects involved so it seemed that the properties/variables specific to a particular project should reside in their own file leaving the kettle.properties file to contain settings common across the various projects.

    In other words, these properties are not related to the USER but to some PROJECT, so the user-based approach to locating the kettle.processing file doesn't work very well. The USER that will be running will be the same for all the projects - so either all the projects have to share the same kettle.properties or you need some command line based mechanism for selecting/setting different property values.

    This approach is good for running transforms 'by-hand' in Spoon (i start Spoon with an appropriate argument and then run my ReadProperties transform once to set the various variables and then i can just run transforms), I can also run jobs.

    Note, , the javascript for creating variables will not work in 3.0.
    Sven has added new javascript functions for managing variables in 3.0, i have plans to add similar functions to 2.5.1 so i can use the same javascript. I don't think i will be affected by the subjob issues.

    I have attached the ktr file. Note, this transform is intended to be run as a step within in a job thus sequencing and separating the 'create variables' transform from the other transforms of the job.

  4. #4
    Join Date
    Jul 2007
    Posts
    247

    Default

    You can define an system environment variable KETTLE_HOME (on Windows Start -> Control Panel -> System -> Advanced -> Environment Variables, on Unix there's something else...) and point it to e.g. your kettle dir. Create a subdirectory named .kettle and store your properties-file there. Now all users on the same machine will use the same property file. This is an easy way when all users use the same server for working with kettle.

    Another approach is the create a parameter file or table in a database that contain all your variables and parameters. Create two transformations: one that defines the variables (set scope as valid in the root job) and a second one that fills your variables with the data from the input source. If you work with 2.5.0 or 2.5.1 there should be no problems setting them using JavaScript.

    BTW: I guess you forgot to post your transformation

    Regards,
    Ben

  5. #5
    Join Date
    May 2007
    Posts
    13

    Default

    sorry, i have attached the transform file ...
    Attached Files Attached Files

  6. #6

    Default

    I couldn't open the transformation, it seem to have errors. Can you please upload it again?

  7. #7
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    There's nothing wrong with the transformation.

    I'm not sure why you insist on jumping through hoops setting the variables in JavaScript though.
    What was wrong with using the "Set Variables" step?

    Java and internal API usage are going to get you in trouble on the next release upgrade.

    Thanks,

    Matt

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.