Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Retrieve data from Web Server

  1. #1

    Question Retrieve data from Web Server

    I have a unique situation here..

    From my kettle job I send an HTTP request to a web server, receive data back and write it to a text file. I am using Java Script to send and receive HTTP requests / responses.

    Since it involves requests in the range of 100,000s, the jobs slow down very much. Here's the logic of what I try to do -

    1) Read rows of data in text file
    2) Send HTTP request for each row read in previous step
    3) Receive HTTp response
    4) Write data received from previous step to text file

    Text File Input --> Java Script (HTTP req/res) --> Text File output

    Is there a way to improve on the performance of the job? Has anyone been in this situation before? If so, I would appreciate to learn from you...

    Thanks in advance!!!!

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    Don't see anything immediately. You're dependent on the http at the other side. And if they do support multi-requests (doing more than 1 request in 1 go) it would take some fidgeting to get around Kettle's limitation of processing rows 1 by 1.

    Regards,
    Sven

  3. #3

    Default

    Sven,

    Do you mean to say that some additional Java code would help process multiple rows at one go - in case I get bundled responses from the Http server?

    Thanks for clarifying.

    Singa

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    1) I don't expect the other side will have multi-requests
    2) If it would have, PDI only does single requests so you would be stuck there
    3) Put if you e.g. would output data in the right format and then use some external program to get multi-requests back it would probably be faster (but only if your server/web app supports it).

    No quick fixes

    Regards,
    Sven

  5. #5
    Join Date
    Aug 2007
    Posts
    3

    Default

    how are you getting javascript to make HTTP requests, all attempts i have tried end in this error.

    Error: JSRULE.ERROR_0003 - Javascript rule execution failed (org.pentaho.plugin.javascript.JavascriptRule)

  6. #6

    Default

    You can use SOAP requests via Java Script. It works for me !!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.