Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Commit Size and Lost Data on PDI

  1. #1
    Join Date
    Mar 2015
    Posts
    5

    Default Commit Size and Lost Data on PDI

    Hai All,

    i have one transformation with multiple process in database for insert and update my row, when i used commit size=1 the with data processed about 30000, it work well and no lost data, but this way is affected time perfomance. if i change my commit size about 50 or 100, it use less time than before, but the problem is some of my data to be process are lost (ex:30000 data, the data in database only about 29900). how to used the right configuration for commit size in my table output & update, anyone happen this problem too?

    thanks

  2. #2
    Join Date
    Mar 2015
    Posts
    190

    Default

    i hope in this case we better to use the option " make the database transactional".

    Which version you are using? have you checked the jvm size in spoon.bat ,.sh files?

  3. #3
    Join Date
    Mar 2015
    Posts
    5

    Default

    iam using pdi-ce-5.3.0.0-213.
    for jvm size PENTAHO_DI_JAVA_OPTIONS="-Xmx1012m" "-XX:MaxPermSize=256m"

  4. #4
    Join Date
    Aug 2015
    Posts
    313

    Default

    Quote Originally Posted by Mazhuri View Post
    for jvm size PENTAHO_DI_JAVA_OPTIONS="-Xmx1012m" "-XX:MaxPermSize=256m"
    I hope it is always better to update size like -Xmx512m * 2. anyhow issue is not with this but we better to follow standards. Have you tried option " make the database transactional" ?.

    You can also increase few more if you are having lot of RAM size in your machine

    Thank you

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.