Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Table Output commit size too large?

  1. #1
    Join Date
    Feb 2010
    Posts
    10

    Question Table Output commit size too large?

    Hi Everyone,

    i checked this forum about my Issue, but i havent found something similiar.

    I have around 60K Records and more, i use the table output step and had a commit size with 100k. I did dis because i could have more then 1Mio. Records and i will not have problems with the oracle archivlogs (full...and so on).....
    But now i havent all Records in the Table....it looks like that the commitsize from 100k, couldnt handle a lower amount of the commitsize, is that true or is it possible that a Memory Issue i have. I get no errors...

    Thanks a lot for the help.

    Cheers
    Alex

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Why don't you set it to 1000 then? A commit size of 100k is probably problematic for the driver and reduces performance (by reducing parallelism in the transformation).

  3. #3
    Join Date
    Mar 2011
    Posts
    257

  4. #4
    Join Date
    Feb 2010
    Posts
    10

    Default

    The Problem was, that we had problems with the archivlogs from the Oracle Database, they run full, because we have often an amount of 10mio Records and then a commit size with 1000 produces to muche logs...and then the hole performance were down...thats why...or im wrong?
    but thnx a lot for this answers...i need this to communicate the problem to the customer...thnx a lot

  5. #5
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    The commit size or batch size is unrelated to the archive logging of the Oracle database.
    The archive log needs to log everything for backup and recovery purpose.
    The only solution there is to move your batch window, process less records in total or increase the archive log size.

    And no, Oracle doesn't support 100k batch sizes, I think the limit is somewhere around 64k but it could be 16k for certain versions.

  6. #6
    Join Date
    Feb 2010
    Posts
    10

    Default

    Matt, thanks a lot for this informations and your help!!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.