Hitachi Vantara Pentaho Community Forums
Results 1 to 8 of 8

Thread: Launch Job Executor Dynamic # of Times

  1. #1
    Join Date
    Jul 2014
    Posts
    10

    Default Launch Job Executor Dynamic # of Times

    I have a transformation that takes rows and for each row I am executing a job via the Job Executor. Right now I'm hardcoding how many rows I'm expecting but I would like to have this be dynamic. How can I do this?

    http://i1173.photobucket.com/albums/...ps7a7e3414.png

  2. #2
    njain111 Guest

    Default

    Instead of hard coding 25, you can use a variable there ${MaxCPUCore}

    You can either put this variable in kettle.properties or make it more dynamic and populate it in a transformation which can execute before this transformation.

  3. #3
    Join Date
    Jul 2014
    Posts
    10

    Default

    Thanks for the reply. I need to get the number of rows that are coming from "Get rows from result" and then use that number as the variable for the # of copies to launch. How would I go about this? Sorry, I'm just getting started w/ PDI.

  4. #4
    njain111 Guest

    Default

    In the transformation where you are 'copying rows to result', before copying rows, I would do a 'group by' to get 'nr of rows' and then set it to variable.

    Make sure to do this as a side stream, so that you set this variable only once.

  5. #5
    Join Date
    Jul 2014
    Posts
    10

    Default

    What do you mean when you say side stream? I tried to do the following but I get the error "Only 1 input row was expected to set the variables and at least 2 were received" (which I'm assuming is what you were referring to).

    http://i1173.photobucket.com/albums/...ps87066024.png

    Thanks again for all your help so far

  6. #6
    njain111 Guest

    Default

    I think you got the side-stream part right here. As group by is not in main-stream flow.

    But there is a bigger problem here ... you cannot set a variable and use it in same transformation.
    I see generate rows instead of 'copy rows here' so, probably you are just testing here with generate rows.
    But, this group by and set variable needs to be in the transformation which copies rows to result.

    To get rid of "Only 1 input row was expected to set the variables and at least 2 were received" this, select nothing in group field (as you have to group all the rows) in aggregates select type='Number of rows' and then set the Name to set variable.

  7. #7
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Regarding the Group By aggregate "Number of values": You don't have to count rows again, when Kettle already did it for you: Just ask for the number using "Output step metrics". Put the two shown steps somewhere into your transformation, no additional hops necessary. Don't worry, when no step names are there to pick from, you can type them right away.

    Name:  OSM.png
Views: 48
Size:  6.3 KB
    So long, and thanks for all the fish.

  8. #8
    njain111 Guest

    Default

    Nifty!

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.