Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Performance

  1. #1

    Default Performance

    I am still having performance issues and would love some help.

    I am running Java 1.4 with 1GB allocated. I am connecting to an Oracle 10g RAC using the ojdbc14 driver.

    I am retrieving 180,000 rows of 11 fields. I am testing with the latest pentaho source code I pulled down this morning.

    My action sequence gets data from MQL and passes a live resultset to a custom class I wrote. The custom class loops through each row and field but does not do anything with the data (fills a local Object variable that is then set to null and reused). I print a status every 10,000 rows so I can get some times.

    Without my Eclipse profiling software running it takes about 8 minutes to process all of the rows. With the profiler running it takes over a minute to do 10,000 rows.

    Why does it take so long?

    getColumnCount() is called repeatedly and I noticed that the Pentaho SQLResultSet class has variable to hold the column count but the next() method does not use it. I think this was an oversight and needs to be fixed.

    Memory continues to increase during the processing. Without profiling I can only process around 1.5 million rows before running out of memory.

    Profiling only tells me most memory is consumed by character and byte arrays.

  2. #2

    Default Was using OCI not JNDI

    I noticed my xmi was pointing to Oracle OCI not my JNDI. it also had some default MYSQL parameters (fetch size = 500)

    I am using JNDI now and bumped up the fetch count to 20000. Much faster now but hit a wall after 1.4 million records. That is when I have used up all of 1 GB.

  3. #3

    Default Scrollable Resultset

    The limit is due to the scrollable resultset. If I switch to forward_only I can process millions of rows without a large increase in memory.

    I created a feature request to allow specifying forward-only in certain scenarios.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.