Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Nested Loop in Job

  1. #1
    Join Date
    Aug 2014
    Posts
    13

    Default Nested Loop in Job

    Hi all. I'm a non-Developer trying to do a nested loop in a Job but can't get it to work correctly. Here's what I need to do:


    1. Call a REST API to get a list of Account records and add the rows to the Result
    2. For each Account record in the Result, call a 2nd REST API and retrieve up to 500 rows
    3. If there are still more rows to retrieve from the 2nd API for the Account record, call the REST API again using the "NEXT PAGE URI" that came back from the previous request
    4. Continue retrieving rows from the 2nd API for the Account record (500 at a time) until there are no more to retrieve
    5. Repeat the process for the next Account record in the Result.


    So far I have a single Job that calls 2 transformations (1 to get the list of records for the result, 1 to retrieve rows for each record in the result). The job is structured to loop for each Transformation if the previous call didn't return all of the rows available. The 2nd transformation in my Job is also set to execute for every input row from the previous transformation. It's not working correctly whenever the 2nd Transformation needs to make multiple calls to retrieve all of the Records for a single Account. I think the problems are from trying to do a loop back to a Transformation that already has the "Execute every input row" option selected.

    Is there a better way to do this? Any help would really be appreciated.

  2. #2
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    REST API Chunking is a common problem, and one that I haven't seen solved cleanly here yet.
    Part of the problem is that you're not really releasing the row sets until you hit the end of the REST data set.

    There are a number of posts on this board on how to do it, but they also complain about out-of-memory issues as well.

  3. #3
    Join Date
    Aug 2014
    Posts
    13

    Default

    Thanks for the reply. I found a Post that mentioned using a Job-SubJob-Transformation configuration, and that seems to work. The Top-Level Job calls the SubJob for every row in the result while the Loop occurs in the SubJob.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.