Hitachi Vantara Pentaho Community Forums
Results 1 to 10 of 10

Thread: Metadata Injection and Variables

  1. #1
    Join Date
    Apr 2008
    Posts
    1,771

    Unhappy Metadata Injection and Variables

    Hi.
    I have some difficulties in a job that I'm running.
    This job has 2 Transformations:
    1. Create Variables
    2. Create multiple files using variables from previous step.

    It would be easy if... I would not try to use the Metadata Injection step.

    Basically, in my second transformation I use a CSV Input step which is injected using the MJ step.
    Therefore, following the sample, I use the Transformation including the Metadata Injection Step instead of my second Transformation.

    The issue is that when I run my job, I have an error stating that my variables are not used (it keeps trying to save a file called {PATH}/{OUT_FILE})

    Any suggestions?
    If needed I can provide more info.

    Mick

  2. #2
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    The error message that I have when running my job is the following:

    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : Error opening new file
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : org.pentaho.di.core.exception.KettleException:
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : java.io.IOException: Error creating output file! Parent directory [C:\Program Files\data-integration\${OUT_PATH}] does not exist.
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : Error creating output file! Parent directory [C:\Program Files\data-integration\${OUT_PATH}] does not exist.
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) :
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : at org.pentaho.di.trans.steps.excelwriter.ExcelWriterStep.copyFile(ExcelWriterStep.java:562)
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : at org.pentaho.di.trans.steps.excelwriter.ExcelWriterStep.prepareNextOutputFile(ExcelWriterStep.java:638)
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : at org.pentaho.di.trans.steps.excelwriter.ExcelWriterStep.init(ExcelWriterStep.java:862)
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : at org.pentaho.di.trans.step.StepInitThread.run(StepInitThread.java:52)
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : at java.lang.Thread.run(Unknown Source)
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : Caused by: java.io.IOException: Error creating output file! Parent directory [C:\Program Files\data-integration\${OUT_PATH}] does not exist.
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : at org.pentaho.di.core.vfs.KettleVFS.getOutputStream(KettleVFS.java:261)
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : at org.pentaho.di.trans.steps.excelwriter.ExcelWriterStep.copyFile(ExcelWriterStep.java:551)
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : ... 4 more
    2011/09/27 09:50:34 - Create Excel Files.0 - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : Couldn't prepare output file ${OUT_PATH}/${TERRITORY}
    2011/09/27 09:50:34 - Create Excel Files.0 - Finished processing (I=0, O=0, R=0, W=0, U=0, E=1)
    2011/09/27 09:50:34 - 07 - Injection - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : Errors detected!
    2011/09/27 09:50:34 - 07 - Injection - ERROR (version 4.2.0-stable, build 15748 from 2011-09-08 13.11.42 by buildguy) : Errors detected!

    Mick

  3. #3
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Are OUT_PATH and TERRITORY parameters?

  4. #4
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    Hi Matt.
    Yes, they are.

    Mick

  5. #5
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    Hi Matt.
    I tested the same scenario and I had the same error.
    I don't know if I'm doing something wrong, but it seems to me that if I use the Matadata Injection transformation, parameters are not passed to its child transformation.

    In my scenario I have:
    Job A which includes:
    Transformation 1->Transformation 2.

    Transformation 1: Create Variables (parameters).
    Transformation 2: Metadata Injection which point to Transformation 3.
    Transformation 3: Create files using Variables (parameters).

    The problem is that Transformation 3 does not recognise Variables (parameters), instead uses Variables as text, therefore my filename is {TERRITORY}.xlsx

    Can anyone confirm this so that I can set up a Jira?

    Thanks.
    Mick

  6. #6

    Smile Dynamic Metadata Injection

    I am surprised that Meta data cannot be derived in some circumstances within an input step. For example, as part of the Text File Input step, the "get fields" options parses the header information from the text file. So I don't understand why this has to be explicitly done in another step? What am I missing here? The same would be true if ingesting data from a data base table. I believe that this would be useful.

    Also I believe it would be nice to have a save-metadata capability from within Spoon. That way you may be able to connect transformations easier.

    Also what I am reading on the internet is that this example doesn't work.

    I ran the user_metainject_step transform and it shows all the steps in the transform excuting. But read_csv_file doesn't execute.

    I would think that there would be step to "Pull metadata from another step". This looks like more of a push meta data to another step.

    How do you execute this example?

  7. #7
    Join Date
    Oct 2007
    Posts
    26

    Default Also problem with parameter sending to Metadata Injection step

    I'm also having a problem with this.
    I use the Metdata Injection step to load files generically, and its works fine.

    But I cannot push the parameter STG_TARGET_TABLE_GENERIC to the injection step, so I set it as a Kettle property.
    This prevents me from loading files in parallel, because the scope of the property is JVM.
    The injection step seems to not 'see' it when the scope is more limited than JVM.

  8. #8

    Default Same problem

    I have also a variable problem in my etl process. The child transfomation does not replace my variables. Is this (now) a know bug? some news?

  9. #9
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    File JIRA cases folks. In a few weeks I'll be picking up all the metadata injection feature requests and bug reports so do it the sooner the better.
    Since things get complex, create simple reproduction examples and attach them to the JIRA cases, otherwise I'll have no choice but to ignore them.

  10. #10
    Join Date
    Jun 2009
    Posts
    22

    Default

    I have had some of the same issues. For my text output steps, I needed parameters. But instead of parameters, I generated a single row and then used the row normalizer to inject single values (i.e. filename). I then used the filename field from a previous step instead of a parameter. That worked for me. Now, I am trying to do the same with a database connection and that's not working as easily.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.