Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Sort step hangs

  1. #1
    Join Date
    Feb 2008
    Posts
    216

    Default Sort step hangs

    Hello -
    I have a transformation where I am doing a merge rows on two inputs. I have a sort step before the merge rows, but for some reason, my sort step starts to hang during processing. One input is 1.4M rows and the other is only 95K rows. The 1.4M rows stops sorting at 696K rows of output and the 95K rows stops sorting at 20K rows of output. I have no idea why this is the case. I've tried increase the number of copies of steps etc. I can't fathom why it is hanging. In other transformations with the same set of rows of input, the sort completes in reasonable time. This one is not erroring or anything, just hanging.

    I am executing this remotely and the server has plenty of disk space (184GB free). Any thoughts?

    *** update ***
    Just wanted to report that I'm seeing the same behavior when I execute locally, too.

    Oh, and I tried caching the sort steps at different sizes. On the 1.4M rows, I started with 25K, changed it to 50K and then changed it to 10K. Sames results with all.
    Last edited by DebbieKat; 08-12-2008 at 06:29 PM.

  2. #2
    Join Date
    Feb 2008
    Posts
    216

    Default Increasing Nr rows in rowset

    I was able to get this to move further along by increasing my Nr rows in rowset configuration. I had to increase it to 100K!!!! That seems awfully high to me to get sorting to work? And, why don't I have this problem with any other transformations? I have never had to increase this value so high before.

  3. #3
    Join Date
    Jul 2007
    Posts
    247

    Default

    Hi Debbie,

    maybe this might help a bit to answer your questions?

    http://wiki.pentaho.com/display/EAI/Performance+tuning


    Regards,
    Ben

  4. #4
    DEinspanjer Guest

    Default

    That sounds odd. Are you seeing any out of memory errors or excessive disk IO from swapping or anything?

  5. #5
    Join Date
    Feb 2008
    Posts
    216

    Default

    Quote Originally Posted by DEinspanjer View Post
    That sounds odd. Are you seeing any out of memory errors or excessive disk IO from swapping or anything?
    No out of memory errors that I've seen. I haven't really been monitoring the IO, though. Again, I have other transformations dealing with just as many records if not more and just as many fields and I've never had to set the rowset nr so high.

  6. #6
    Join Date
    Feb 2008
    Posts
    216

    Default

    Quote Originally Posted by BeLienig View Post
    Hi Debbie,

    maybe this might help a bit to answer your questions?

    http://wiki.pentaho.com/display/EAI/Performance+tuning


    Regards,
    Ben
    This sort of discusses what I'm seeing, but it is curious that the wiki mentions this behavior is fixed in later versions and I'm using 3.0.4. And I did leave the default checked for Manage Thread Priorities. Seems like curious behavior anyway.

  7. #7
    Join Date
    Feb 2008
    Posts
    216

    Default Could database latency cause this behavior?

    I am wondering if we are seeing more latency on our database recently and that might be the cause for having to up the Nr of Rows in Rowset setting. Could it be that latency is causing more locks to be created and causing buffer issues or something? If I increase this setting then it is waiting longer before it commits, right?

    Am I barking up the right tree here?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.