Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Calling REST AIP with pagination

  1. #1
    Join Date
    Jan 2018
    Posts
    2

    Default Calling REST AIP with pagination

    I have a REST api call to extract data periodically.
    The API has pagination, one time return back 50 rows or something.

    Let say I'm calling http://www.blabla.com/pageid=1

    The return format is in json like below
    {
    "data" :
    [
    {"field1": "field value", "field2": "field2 value"},
    {"field1": "field value", "field2": "field2 value"},
    {"field1": "field value", "field2": "field2 value"},
    ..
    ]
    ,
    "total" : 80
    "prev_url": "http://www.blabla.com/pageid=1",
    "next_url": "http://www.blabla.com/pageid=3"
    }


    I need to repeat the REST call till the next_url value is empty.
    How can I achieve that in PDI?


    Any suggestion is appreciated.
    Thanks.

  2. #2
    Join Date
    Jan 2018
    Posts
    2

    Default

    Implemented the solution and it is working.
    1 Job, 2 transformation


    In the Job
    - Execute 1st Transformation step
    - Simple Evaluation step - using the next_url value is blank
    - if blank execute Success step
    - else Execute 2nd Transformation step and hop to Dummy step to hop back to Simple Evaluation step
    In 1st Transformation
    - CallREST API with url specified.
    - Extract the next_url from return json result
    - * Call "Copy rows to result" which include the next_url


    In 2nd Transformation
    - * Call "Get rows from result" to get the next_url value from previous step (transformation step in job)
    - Call REST API with next_url value
    - Extract the next_url from return json result
    - * Call "Copy rows to result" which include the next_url

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.