Table Output commit size too large?

[aljechin1973]

Hi Everyone,

i checked this forum about my Issue, but i havent found something similiar.

I have around 60K Records and more, i use the table output step and had a commit size with 100k. I did dis because i could have more then 1Mio. Records and i will not have problems with the oracle archivlogs (full...and so on).....
But now i havent all Records in the Table....it looks like that the commitsize from 100k, couldnt handle a lower amount of the commitsize, is that true or is it possible that a Memory Issue i have. I get no errors...

Thanks a lot for the help.

Cheers
Alex

[MattCasters]

Why don't you set it to 1000 then? A commit size of 100k is probably problematic for the driver and reduces performance (by reducing parallelism in the transformation).

[hansva]

it might be related to http://jira.pentaho.com/browse/PDI-7586

[aljechin1973]

The Problem was, that we had problems with the archivlogs from the Oracle Database, they run full, because we have often an amount of 10mio Records and then a commit size with 1000 produces to muche logs...and then the hole performance were down...thats why...or im wrong?
but thnx a lot for this answers...i need this to communicate the problem to the customer...thnx a lot

[MattCasters]

The commit size or batch size is unrelated to the archive logging of the Oracle database.
The archive log needs to log everything for backup and recovery purpose.
The only solution there is to move your batch window, process less records in total or increase the archive log size.

And no, Oracle doesn't support 100k batch sizes, I think the limit is somewhere around 64k but it could be 16k for certain versions.

[aljechin1973]

Matt, thanks a lot for this informations and your help!!