PDA

View Full Version : Extraction and Conversion speeds?



rider555
02-23-2007, 04:09 AM
We're trying to use Spoon for our ETL projects, but we noted that different PCs yield different speeds to convert flat files containing 300k records into Oracle. Commit size is already at 15000, but the time elapsed is at 2mins 20secs. Other ETLs systems does the same job within 34secs. What am I doing wrong? Please help....

sboden
02-25-2007, 04:42 PM
You'd probably better switch to the "Pentaho Data Integration" forum for your PDI questions (same community forum, but a different section).

For your timings... 34 seconds would be 9000 rows per second, 2:20min would be 2000 rows per second. The 2000 seem reasonable to me, having some experience with other ETL tools.

- What do you get as timing when your the ETL on the same server as your other ETL is running?
- You can start duplicate copies of your output step (right click and then search the menu), if the rows won't interfere with each other.
- An Oracle bulkloader is on the way as a step, but it will take me a few more weeks to complete it to 100%.
- If you would be using OWB to process in 1 database your 9000 rows would be the fastest you'll get. Other ETL tools should be in the Kettle range.

Regards,
Sven