Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Extraction and Conversion speeds?

  1. #1

    Default Extraction and Conversion speeds?

    We're trying to use Spoon for our ETL projects, but we noted that different PCs yield different speeds to convert flat files containing 300k records into Oracle. Commit size is already at 15000, but the time elapsed is at 2mins 20secs. Other ETLs systems does the same job within 34secs. What am I doing wrong? Please help....
    It's Morphin Time!!! Busters ready GO!!!

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    You'd probably better switch to the "Pentaho Data Integration" forum for your PDI questions (same community forum, but a different section).

    For your timings... 34 seconds would be 9000 rows per second, 2:20min would be 2000 rows per second. The 2000 seem reasonable to me, having some experience with other ETL tools.

    - What do you get as timing when your the ETL on the same server as your other ETL is running?
    - You can start duplicate copies of your output step (right click and then search the menu), if the rows won't interfere with each other.
    - An Oracle bulkloader is on the way as a step, but it will take me a few more weeks to complete it to 100%.
    - If you would be using OWB to process in 1 database your 9000 rows would be the fastest you'll get. Other ETL tools should be in the Kettle range.

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.