Hitachi Vantara Pentaho Community Forums
Results 1 to 8 of 8

Thread: Jobs and Transformations

  1. #1
    Join Date
    Mar 2009
    Posts
    8

    Default Jobs and Transformations

    Here's my situation:

    I've got a text file with a list of urls in it. For each url in the text file, I want to HTTP download the file. The problem is...

    1. HTTP step is in Job
    2. file input steps are in Transformation

    So, inside a transformation, I can open the text file and get the URL's out. How do I process those URL's 1 by 1 in the job and download the file? I can't seem to call a job from a transformation.

    -Greg

  2. #2
    Join Date
    Nov 2009
    Posts
    688

    Default

    In the transformation you set a variable with the url and use that variable in the http step.

  3. #3
    Join Date
    Apr 2009
    Posts
    21

    Default

    You can probably use checkbox 'Run for every result row' in Transfer a file using HTTP.

    Create a transformation reading the list of url; end with a Copy rows to result
    Create a Job calling this transformation and follow it up with Transfer a file using HTT checking the box.

    let us know if it works...

    Thijs

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Maybe I'm a bit simple minded, but:

    Quote Originally Posted by ggreaves View Post
    The problem is...

    1. HTTP step is in Job
    Sure, there is an HTTP entry for the job, but there is an HTTP Client step as well, so you could easily wire your URL list to it in a transformation.
    Attached Files Attached Files
    So long, and thanks for all the fish.

  5. #5
    Join Date
    Mar 2009
    Posts
    8

    Default

    This does work. However, there's a problem. The HTTP step has a target file that only accepts variables (not field names). Since I want every file to have a unique name, what should I do? I tried appending date / time to the target file name but it happens so fast that instead of 4 files, I only get 2 because the time hasn't changed enough.

  6. #6
    Join Date
    Mar 2009
    Posts
    8

    Default

    This should work. However, the result is actually an .xls file. How do I save this result. I've tried outputting as a text file, with only one field (result), but it doesn't come up in excel.

  7. #7
    Join Date
    Mar 2009
    Posts
    8

    Default

    It's not just one url. It's a collection of urls. I need to ftp for every url in the input file. a variable won't work, unless I'm missing something.

  8. #8
    Join Date
    Mar 2009
    Posts
    8

    Default

    how long does comment moderation usually take? I posted replies about 9 hours ago.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.