Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Can You Delay/Stagger HTTP Post step requests?

  1. #1

    Default Can You Delay/Stagger HTTP Post step requests?

    I have what I would think is a pretty common problem. I'm calling a service with HTTP Post and getting responses back. This all works fine except when I get many records it's pummeling the server with too many requests which it subsequently starts returning errors since it's being overloaded past the allowable limits of the service.

    Is there a way to slow it down or make it wait for a request to come back before it sends another one? I know this is bad for performance but it's something I need to solve to make kettle work for our data integration. Any help would be very appreciated. Thanks!

    Walter.

  2. #2
    Join Date
    Nov 2008
    Posts
    777

    Default

    There is a Delay Row step that can be inserted in order to slow down the processing of the rows by a fixed time value.

    Another option would be to build up the rows in a job and then execute them (using the "execute for each row" option) in a transformation that handles one row at a time. Using that method, each request would complete before another request was sent.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  3. #3

    Default

    Quote Originally Posted by darrell.nelson View Post
    There is a Delay Row step that can be inserted in order to slow down the processing of the rows by a fixed time value.

    Another option would be to build up the rows in a job and then execute them (using the "execute for each row" option) in a transformation that handles one row at a time. Using that method, each request would complete before another request was sent.
    Man, thanks so much. That is 'exactly' what I needed. I knew something had to be there. Thanks again!

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.