Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Dimension Lookup/Update very slow, what are the alternatives?

  1. #1

    Question Dimension Lookup/Update very slow, what are the alternatives?

    My Dimension Lookup/Update is about 10x to 20x slower than the regular Table Output.

    What are some general common ways to speed this up?

  2. #2

    Default

    Try to use "Table Output" step and enable "Error Handling" to manipulate duplicates

  3. #3
    Join Date
    May 2014
    Posts
    4

    Default

    You can pre-process your batch and pre-load/update all your new dimension data. I generally will read in all of the batch data, then do a GroupBy on the dimension field. Send that output to a database lookup step (cache data set to "load all data from table"). If no row returned from lookup then send the error output to a table output step or upsert step that will create/update the dimension. Then you can run your regular transformation using just a database lookup step knowing that all of your dimensions are already processed.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.