Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: REST lookup or create in PDI

  1. #1

    Default REST lookup or create in PDI

    Hey,


    I'm facing a challenge which I would appreciate some help finding an idiomatic and perfomant solution for.


    I have a transformation which is like this:


    CSV -> Transform -> REST Call


    Csv looks like this:


    sku;desc;price;catName


    And the required JSON to the REST call looks like:


    [
    {
    "sku": 42342,
    "description": "some descr",
    "price": 245,
    "catId": 7890
    }
    ]


    So far so good, and the initial tranformation works well! However, I need to do a lookup to transform the catName (category name) to a catId (category id) and if the category doesn't exist, create one and use the new ID. This look up would happen via a REST call too. There could be 100k+ input rows and with my current solution I would need to have 100k+ REST lookup calls, which is going to kill performance. There will be around 2-10k categories.
    Im looking for a more performant approach to doing the category transformation (lookup or create).


    Any advice/pointers/links would be more than appreciated!


    Thanks,


    -David

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Can't you retrieve the categories first and then use a Stream-Lookup?
    So long, and thanks for all the fish.

  3. #3
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Quote Originally Posted by marabu View Post
    Can't you retrieve the categories first and then use a Stream-Lookup?
    Continuing on this idea...

    Copy your rows into 2 parallel streams coming out of the Text File Input step.
    Select only the Category Names and Sort them, passing on only unique rows.
    Now pass this on to your "Category Name -> Category ID" REST function (which will return either the existing Id for that Name, or if the name doesn't exist, will return a new ID)
    You should now have a list of Category Names and IDs to feed into the Stream-Lookup that Marabu suggested.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.