Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: merge join hanging 3.0.3/3.1

  1. #1
    Join Date
    Mar 2008
    Posts
    13

    Default merge join hanging 3.0.3/3.1

    I'm reading in data and copy the flow into 2 flows and try to rejoin each flow back together in a merge join. One side doing a lot of filter, grouping, etc. The other side almost straight into the merge join. Works ok on small volume, but if i get over 20k row - its just hanges. Is it possible there is a timing issue with merge join? Any thoughts why it might hang on still relatiively light volume.
    thanks
    -shawn

  2. #2
    DEinspanjer Guest

    Default

    Have you been monitoring the memory usage of the JVM while this transformation is running?

  3. #3
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    It's probably not an issue with the Merge Join. I bet it's that you are running out of buffer space.
    The transformation stalls because you are basically joining data with itself.
    Either double the source step itself or put a Sort/Blocker step in between to buffer the rows.

  4. #4
    Join Date
    Mar 2008
    Posts
    13

    Default

    We tried the extra sort step which worked. I didn't look into JVM usage to see about the memory space. I wiil have to check this out to avoid future problems. I need to read up a little on the JVM memory usage. thanks for the advice.
    -shawn

  5. #5
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    It's not running out of memory, it's running out of buffer space. You can change the buffer space (Row set size) in the transformation settings (Misc tab).

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.