Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: return code = 2 ; while doing lookups on the dimension table to load the datamart.

  1. #1
    Join Date
    Jun 2010
    Posts
    15

    Default return code = 2 ; while doing lookups on the dimension table to load the datamart.

    Hi Experts,

    I am facing an error in Pentaho Data Integration while doing lookups on the dimension tables to load the datamart. I have no transformations on the data, only resolving dimension keys and loading the fact.

    The KTR will execute fine, if the fact records is less than 75,000. But if it’s more than 75,000 I get an error with return code = 2 ( java heap memory issue ).
    Apart from one dimension table which contains about 700,000 records, all other dimension tables are very small (less than 3000).

    The lookups are done one by one. ie. I lookup on one dimension and then move on to the next.
    My commit size for each is 10,000 records. I have enabled cache and unchecked ‘load all data from table’


    The flow is as below.
    Load the flat file (CSV) into a staging table
    Resolve the keys to the dimension (9 dimensions) [one by one]
    Filter records that don’t match the dimensions (one by one)
    Load the matching records to fact
    Load the unmatched records to a reject table

    Please let me know what are the steps I need to take to handle big loads.

    Thanks ,
    AMR

  2. #2
    Join Date
    Aug 2008
    Posts
    563

    Default

    return code = 2 ( java heap memory issue ).
    Increase the amount of assigned memory in spoon.bat or spoon.sh (depending on your OS). Also, if this doesn't help, in the database lookup step disable the option to load all data into the cache if you set it. If you enabled cache, set a limit as well (your transformation will be way slower than, but it will work).

    The lookups are done one by one
    Did you set the transformation to transactional in the transformation settings? Otherwise, everything will run in parallel if all the steps are in the same transformation.
    Best regards,
    Diethard
    ===============
    Visit my Pentaho blog which offers some tutorials mainly on Kettle, Report Designer and Mondrian
    ===============

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.