Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: How Do I calculate "Copy rows to result" memory limit?

  1. #1

    Question How Do I calculate "Copy rows to result" memory limit?

    Hi Guys,

    Is there anyone that can help me determine the memory requirements for "copy rows to result"?

    I have a job that calls a transformation that merges two database tables and split the result to three different streams. The largest stream (roughly 100MB) I want to copy to the next transformation in order to process it separately and save me having to read the merged rows from the table again.

    I have set my heap space to OPT=-Xmx1024m but it still fails with the heap space error. It fails despite the (job) log indicating that all transformation stages (including the "copy row to result") have finished successfully, i.e. it crashes just before you expect the next transformation in the job is about to start.

    I would have thought 1GB would have been sufficient to handle the merged data set.

    How much memory of the 1GB is available for data?

    When doing the calculation how much space is tkane up by a string[222] with only 10 characters in it?

    Am I on the right track?

    Regards
    Paul

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    If you have to guess, maybe it's better to drop the data to a file or a database table.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.