Hitachi Vantara Pentaho Community Forums
Results 1 to 9 of 9

Thread: Problems with the Resource Exporter

  1. #1

    Angry Problems with the Resource Exporter

    Hi, I have a job that has several transformations.

    In the job I do a 'Check if file exists step', and in several of the transformations I read from an XML file with the 'Get Data from XML step'. The job and transformations are stored in a repository.

    I used the 'Export all linked resources to XML' and got a .zip file with all the resources.

    The problem that I am having is that during the export, the file paths to the input files are not being replaced by the DATA_PATH_x as they are supposed to according to: http://www.ibridge.be/?p=159

    When I open the zip file and look at the .ktr and .ktj I see that the DATA_PATH_x variable was created:
    <parameters>
    <parameter>
    <name>DATA_PATH_1</name>
    <default_value>/Pentaho/BFE_Data_Warehouse</default_value>
    <description>Data file path discovered during export</description>
    </parameter>

    But it is not used in the steps:
    <entry>
    <name>Config File Exists ?</name>
    <description>File Exists</description>
    <type>FILE_EXISTS</type>
    <filename>file:///C:/Pentaho/ BFE_Data_Warehouse/config.xml</filename>

    I would expect to see something like:
    <filename>${DATA_PATH_1}config.xml</filename>
    but this is not the case...

    So obviously, when I move the .zip file to a server and try to run it with:
    ./kitchen.sh -file='zip:file:///home/ccastro/data-integration/BFE_Data_Warehouse/BFE_Data_Warehouse.zip!Job___Data_Warehouse_001.kjb' -param: DATA_PATH_1="./BFE_Data_Warehouse" -level=basic

    It fails saying that the:
    Following required files are missing file:///Pentaho/BFE_Data_Warehouse/config.xml

    Has anyone experienced this behavior? Is there a work around? Should I just define the path variable myself ?

    Thanks,

    Carlos Castro

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    1) File a JIRA bug report (link to this thread)
    2) Apply your work-around :-)

  3. #3

    Default

    I figured...

    I did some more tests on this, and I believe that it is those two steps that have issues when exported. Other 'file related steps' seem to work fine. More specifically here is what I found:

    'Check if File exists' job step does not replace anything in the filename. So for example if you had a variable or a relative path it leaves them as they are.

    'Get Data from XML step' transformation step does replace the filename with the default value of the of the DATA_PATH_x parameter. This is even worse, as it kills my workaround of using a variable, because it replaces it!!

    Thanks for your prompt respond Matt. I will file the bug report.

    Thanks,

    Carlos

  4. #4

    Default

    I also found this rather annoying problem.

    In the current 3.2 stable release, all the variables in my jobs and transformations are replaced with DATA_PATH and in some instances, hard coded paths from my Windows XP development box (not useful since the deployment environment is Linux).

    Here's what I did to work around this:

    1. Uzipped the linked resources zip file
    2. Opened each job and transformation in the design tool.
    3. Saved each in the same folder as the unzipped files with a slightly different name (I added an "X" to the name in order to keep track of things)
    4. Opened each pair of files in the Eclipse compare view (any diffing tool would suffice, though)
    5. Moved all the variables back and removed all the parameters
    6. Deleted the files with X in the name (non linked resource files)
    7. Zipped the files back up.

    If there was a bug filed, please post up the issue number.

    EDIT: A better work-around is to use PDI 3.2 RC1 to "export all linked resources".
    Last edited by JeffBeard; 02-27-2010 at 08:30 PM. Reason: Updated information

  5. #5

    Default

    OK, I created one myself:

    http://jira.pentaho.com/browse/PDI-2579

  6. #6
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    This is the one from Carlos:

    http://jira.pentaho.com/browse/PDI-2577

  7. #7

    Default

    Thanks. I marked mine as a dup.

    --Jeff

  8. #8

    Default

    I added a comment to the bug since this is happening on other steps as well...

    1. Table Input (reference to db port only.... at least for MySQL, other variables get substituted)
    2. Ping a Host
    3. Simple evaluation

    It's related to the resource exporter because I can export a single transformation to xml and get it to run and have no problems with variables subs. Strange because the uncompressed files are identical.

    On Linux, a good work-around will likely be a series of sed commands to do var subs.

    1. Uncompress files
    2. Run sed commands
    3. zip up
    4. version control
    5. deploy

    Or, do your ETL from a PDI repository.

  9. #9
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Thanks Tom

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.