Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: REST web service pagination

  1. #1
    Join Date
    Jul 2012
    Posts
    6

    Default REST web service pagination

    Hello,

    is there any way to paginate through RESTful web services?
    E.g. if you take the following URL:
    http://example.com/creative?start_element=0&num_elements=100

    Assume the request responds with a JSON message that contains the number of total available elements (among the actual data payload).
    Let's say we used this to find out that we need a total of three loops.

    How do we set up PDI to loop this request (with modified start_element parameter) two more times?

  2. #2
    Join Date
    Jul 2012
    Posts
    6

    Default

    To elaborate, I already looked at this post which mentions three options for looping: http://vrubleuski.blogspot.de/2010/0...in-kettle.html


    Option 1 is "Execute for every row". I can't do that because I don't know ahead how many rows there will be.

    Option 2 is "Circling hops in a job". If I understand correctly, this isn't possible with recent PDI versions anymore.

    Option 3 is using the repeat flag in the start step. I tried doing this, but since my setup is a bit more complex (job in job) this doesn't work as expected (see here for a post about this problem: http://forums.pentaho.com/showthread...-gory-details-) )

  3. #3
    Join Date
    Jul 2012
    Posts
    6

    Default

    Nevermind, I worked around this problem by introducing an additional transformation that calls the webservice just for retrieving the total amount of items.
    Slows down the process, but then again it just takes a few extra seconds.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.