Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Mysql DB with PDI CE 5.3 java 1.8 unable to load 3900000 records

  1. #1
    Join Date
    Mar 2015
    Posts
    190

    Default Mysql DB with PDI CE 5.3 java 1.8 unable to load 3900000 records

    Hi,
    mysql database with PDI CE 5.3 java 1.8 with jvm size(6gb)

    I am trying to load data 3900000 records with in the same database using different scenarios because of to avoid java heap space issue. for that i have used Java script logic to iterate the job for every limit range(1 , 100000), attached java script (wh.txt) as well, could you please suggest best approach.

    scenario 1: Table Input -> Insert/Update
    in this case job is iterating but flailing at 3400000 records (java heap space issue)

    scenario 2: Table input -> Table Output
    in this case , not able to iterate the job every time it is taking limit range(1 , 100000), seems to be java script logic will not iterate at Table output step.

    scenario 3: Table Input ->(few logic) -> synchronization after merge
    in this case also job is iterating but flailing at 3400000 records (java heap space issue)
    wh.txt
    Name:  insert update.png
Views: 35
Size:  9.8 KB
    Name:  Sync merge.jpg
Views: 32
Size:  13.0 KB
    Last edited by ranala; 07-06-2015 at 11:30 AM.

  2. #2
    Join Date
    Mar 2015
    Posts
    190

    Default

    after applying Indexes on few columns and in few transformation modified from insert/update step to update then my jobs are running fine, quite happy but need to look deeply to apply changes for performance improvement.

    Could you please suggest someone any more changes.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.