Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: run stats and performance

  1. #1

    Default run stats and performance

    I have made a very simple transformation: Input Table -> Dimension Lookup/Update. Input table has about 50K rows. When I run it I see that first node reads records very fast upto about 10K rows then slows down at the rate of second node outputing the data (obviously lookups are much slower). Is there a reason for that? I have set up second node with unlimited cache, but my guess that its the first node that would need to cache it all, right (although it doesn' have the settings for that)?

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    You can think of hops as "buffers" between steps... press ctrl-T when in a transformation, go the miscellaneous tab and you will see the default of 10K rows.

    In a transformation that consist of sequentially linked steps the eventual speed will be the throughput of the slowest component.

    In the beginning table input will go fast as it's filling up the hop buffer, but when that's full it will start going at the speed of the dimension lookup.

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.