Problem with Data Load Transformation Memory??

[lazius]

Hi there, there is a transformation that loads each week around 80 text files with around 20 lines each one, into an Oracle database. The problem resides when sometimes they can be loaded without problem, but sometimes they need to be loaded 1 one by 1 or 10 by 10 instead of all at the same time. I haven't seen the log for an error because is at another part far away from me and i was thinking if this can be caused because a cache memory limit or some similar property?

Thanks for your help and your replies!

[pstoellberger]

i don't think that kettle has a problem with loading 80 files
without a log / detailed error message its quite hard to understand your problem

[dhartford]

That sounds like a load-control problem, maybe you want to create something to manage the load going into the ETL process (i.e. serial to do one at a time, or parrallel for only 5 or 10 at a time).

Load-control is outside of kettle, what job step/tool are you using to load the files for ETL processing?

Edit: 80 x 20 is 1600 lines, that is nothing - maybe you are running with default memory settings without setting -Xms/Xms settings, or pulling data from another datasource into memory in your ETL process? Again, nothing really shows what you are trying to do.

[lazius]

Thanks for the reply, I'll try to get the log to see if there's an error but mainly when I try to load for example, 50 text files... 25 files are loaded correctly into the database, but the other 25 files just loads the header not the information. But if you do it 10 by 10, everything loads correctly.. maybe the cause is what you are talking about, the defaukt memory setting and i would be better to use the -Xms, but how can I use that and what's the difference between the usage with the default memory?

I'll try to post the log as soon as I can, guess the next week.

[fabianS]

Default behaviour is 512 MB of memory. This is usually sufficient for the low line count you're suggesting.

Can you verify that problem during testing in Spoon?
I usually set PDI to ignore any header lines - how do you know the header is read and the rest is ignored?