Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Optimizing Pentaho Performance

  1. #1
    Join Date
    Jun 2012
    Posts
    2

    Default Optimizing Pentaho Performance

    Hi all, I had a problem with pentaho recently : I had a collection of data which is stored in MS SQL Server 2008. The data was structured as a relational database where there's one fact table and several dimension tables. The thing is that the fact table was populated with almost 40 millions of record which totally costed almost 13 gb of data. When I tried to load the data to the Pentaho ( using the 'create new data source and then clicked the new analysis menu) I noticed that the SQL Server ( or the Pentaho, not sure at all) took almost all the physical memeory of my system ( 8 GB).

    The question is, how can I by any means to reduce the consumption of memory? or is there any method which is far more efficient to load data to the Pentaho?

    Any help will be appreciated. Thank You.

  2. #2

    Default

    I think there's no other option than upgrading the memory of the server or adapting the query. do you really need all of the 40 mio records or is there a possiblity to limit the count of records with the sql-statement

  3. #3
    nbaker Guest

    Default

    I would see what's going on under load using JConsole. That'll give you an idea of memory usage, object allocation, etc.

  4. #4
    Join Date
    Nov 2005
    Posts
    164

    Default

    For creating analysis datasources with large underlying tables, I still suggest using schema workbench to create your model and publish it to the server.

  5. #5
    Join Date
    Mar 2011
    Posts
    257

    Default

    +1 on what bhagan said !
    not sure what the online schema creator does to see what kind of fields/data there is...

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.