Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: SubTransforms with Varying Number of Stream Fields

  1. #1

    Default SubTransforms with Varying Number of Stream Fields

    Suppose that I have 3 kettle transforms - process_main_table, process_backup_table, and common_sub_transform logic.

    In the process_main_table transform, I have a stream with 8 values.
    In the process_backup_table transform, I have a stream with 15 values (8 shared values which are identical to the process_main_table and 7 other values).
    Both transforms call the common_sub_transform logic.

    When I send the stream into the common_sub_transform, I always get 8 output values (it truncates/removes the other 7 values from the stream).

    Is there an option that I'm missing to retain extra fields when calling a sub-transform? Or is this expected behavior?

    My common_sub_transform does introduce some extra fields during its processing and it uses a select fields step to remove its extra fields. Is that select fields truncating the other values? If so, how do I remove the extra temp fields without also removing the fields from the parent transform?

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    When playing with the demo in the Kettle samples folder, I can't reproduce your problem.
    If you attach your own demo, don't forget to replace Table Input steps by Data grids.
    So long, and thanks for all the fish.

  3. #3

    Default

    Thanks for the reply. I've attached two sample files (sample_parent.ktr and sample_sub_transform.ktr).

    In my attached example, I would like column 'Third' to be available after calling the sample_sub_transform. Not all streams calling the sample_sub_transform will have the 'Third' column. I suppose that I could add all possible fields to the sub-transform, but that would be a sizable number of columns in my actual example and none of the extra columns are modified by the sub-transform at all.
    Attached Files Attached Files
    Last edited by birtPentahoUser8; 04-11-2013 at 11:06 AM.

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    In your subtransformation you added a "Select Values" step.
    Obviously, you wanted to rename the calculated fields, but there's a catch: This tab is primarily about selection, renaming is an additional function.
    So you end up with two fields.

    I made two subtle changes to meet your requirements:

    1. I introduced a "Set field value" step to copy the field values instead of renaming the fields.
    2. I use the "Select values" step to remove unwanted fields.

    In fact I changed some other things, too, but nothing had to do with your problem.

    BTW: A "User Defined Java Expression" instead of "Calculator" would have allowed you to replace the mapped fields with the calculation results in one fell swoop, thus eliminating the two steps mentioned above.

    Have fun
    Attached Files Attached Files
    So long, and thanks for all the fish.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.