Hitachi Vantara Pentaho Community Forums
Results 1 to 10 of 10

Thread: How to dynamically Pass File Name in TextFileInput Step

  1. #1

    Question How to dynamically Pass File Name in TextFileInput Step

    I am invoking transformations through my java application. How do I pass the file name in the TextFileInput. The file name is decided dynamically.
    I saw couple of postings in this forum but doesn't seem to address my problem.

    There is one suggestion about using Injector but seems that using injector I need to parse the input file myself and pass the data to trans.

    Other suggestion is to use variable. How do I dynamically set the value of variable and what is scope of the variable? I would like to restrict its scope so that if there is another instance of transformation running in same JVM that variable's value doesn't get overwritten.

    Thank You.



    -San

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    In 3.0 you can inject variables into a transformation. In 2.5 there are some small problems with variables, and if you use them in a hierachy of jobs and transformations it probably won't work... if you want to give it a try search for KettleVariables in this forum and in the source code (a hint: KettleVariables is no longer around in 3.0)

    Regards,
    Sven

  3. #3
    Join Date
    Jun 2007
    Posts
    138

    Default Dynamic sized List /Array

    Is it possible to pass an Array or list to kettle (2.5 & 3.0) whose size is not fix.Like It is dynamic...

    That means our array will have some x number of Variables,But X is not a fix & cant be pre-decided...
    Regards,
    kedar.mehta@tcs.com ,
    Tata consultancies Ltd

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    In 3.0 by putting them in properties object and injecting that. In 2.5 by trying to push variables 1 by 1 in the environment, but as above with complex job setup it probably won't work all of the time.

    Regards,
    Sven

  5. #5
    Join Date
    Jun 2007
    Posts
    138

    Default Pusing variables

    Pushing varibles,I cudnt get u..
    Did u mean that the variable which i pass from JAVA one by one to kettle?Then what if,I have 1000 of variables...!!!
    I think I cudnt get u...
    Regards,
    kedar.mehta@tcs.com ,
    Tata consultancies Ltd

  6. #6
    Join Date
    May 2006
    Posts
    4,882

    Default

    If you call a transformation from java (in 2.5.x) you can do something as:

    KettleVariables variables = KettleVariables.getInstance();

    And then with variables you can setVariable() the variable-name and its value (http://kettle.pentaho.org/svn/Kettle...Variables.java), but it's tricky to get it working right.

    In 3.0 it works in a different way.

    Regards,
    Sven

    P.S. The above is on variables, not in input data.

  7. #7
    Join Date
    Jun 2007
    Posts
    138

    Default Receiveing Variables in kettle

    Thanx sven..

    Its really tricky but good solution(2.5)..Bt I m confused over one task.As my requirement,Data is dynamic.Means i will come to knw abt size of ArrayList or List on runtime only.

    As per u suggested,I shall create some kettle variables-Array in Java.Bt how kettle will receive them..?

    Means in Kettle if I receive them,
    from get variable tab....How cud i manage that arraylist in javascript.

    If i want to treat all the elements of array,then they have to be declared.Declarating numbers isnt possible..B caz thats some wha Dynamic!!

    Can u put some light over this issue...?!!Thanx in advance...
    Regards,
    kedar.mehta@tcs.com ,
    Tata consultancies Ltd

  8. #8
    Join Date
    May 2006
    Posts
    4,882

    Default

    In Java you can have an array of keys/values and set the one by one up as Kettle variable in a transformation... but you would have single variables as a result in e.g. a javascript step. Which means that if you want to use them in a transformation you would have to know the names of the variables, which would kind contradicts your "dynamicity".

    Forget this one... back to your original problem.

    You want to inject filenames from java in a transformation that you also embed in java: have your filenames in an arraylist or an array in your own java appliication. Iterate over the array. pick the filename and set it up using KettleVariables (in 2.5), execute the transformation, and repeat until you have processed all files. The assumption would then be that your text input step uses this one known name to setup its input.
    Alternatively if you make the the whole transformation java you can get at the text input step and put in there any filenames array that you want to process.

    Regards,
    Sven

  9. #9
    Join Date
    Jun 2007
    Posts
    138

    Default JavaScript & variables

    Wooof...

    U have solutions for all the problems..

    But,
    First option wud require me,to call Kettle trasformation as many as files available.It s better (in terms of,Performance) if Kettle make such decision based on variables given. (just think,what if i have 10 varables to pass..calling kettle from JAVA{JSF-Spring-Hibernate Architect} takes significant time...)

    "Passing an Array" option suits better.Indeed we are passing a MAP.So we just need to configure our methods accordingly.

    But how will i get the elements of array in Javascipt tab?Means I can access them as
    query0=dest[0];
    query1=dest[1];

    So on...But here comes the problem.I know that (for the perticular trasformation) I have such 10 elements in an array.
    SO in Javascript tag,below side in fields I need to define all such variables.Like query1,qery2.upto query9 variables.

    Fine,It will work.But what if 2nd Execution,I have 20 fields?I will be knowing (at run time) that I got 20 variables in array.But My Javascript doesnt know that...

    So,What can be done..

    "Dynamicity " is some thing which hurts me a lot...
    Regards,
    kedar.mehta@tcs.com ,
    Tata consultancies Ltd

  10. #10
    Join Date
    Jun 2007
    Posts
    138

    Default Other suggestions

    I feel,Its bit Odd\tough thing ,what I am Expecting from kettle.. I think,I ll have to try other options which you suggested me for achieving this kind of Dynamicity...

    Thanx,again..
    Regards,
    kedar.mehta@tcs.com ,
    Tata consultancies Ltd

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.