Hi Kettle devs,

Good news keeps coming back from the 3.0 frontline as more steps are being
ported.

I did the Text File Input well as the AddSequence step.
Sven also ported the AddConstants and Abort steps.

We wrote unit tests that now automatically run these new steps against old
and new code to compare speed.
That makes it somewhat easier to provide the results below...

I've tried my best to find a combination of steps that might result in a
slower performance, but so far couldn't come up with anything.
Around 20% performance gain seems to be the minimum so far with
exceptionally up to an 800% increase in performance.
In general you can say that we're reducing overhead here and that
transformations that do heavy computations (type conversions, etc) are less
suspectible to performance gain.
All the other on the other hand...

All the best,

Matt
____________________________________________
Matt Casters, Chief Data Integration
Pentaho, Open Source Business Intelligence
http://www.pentaho.org <http://www.pentaho.org/> -- mcasters (AT) pentaho (DOT) org
Tel. +32 (0) 486 97 29 37



TEXT FILE INPUT

Name of transformation: Text File Input - CSV
Transformation description: Read a large CSV file
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 100.000, runtime: 16,44s, speed: 6.083 rows/s
V3 results, rows: 100.000, runtime: 13,27s, speed: 7.538 rows/s
V3 / V2 = x1,24

Name of transformation: Text File Input - CSV - Strings only
Transformation description: Read a large CSV file, all fields as Strings (no
conversions)
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 100.000, runtime: 6,86s, speed: 14.577 rows/s
V3 results, rows: 100.000, runtime: 4,06s, speed: 24.612 rows/s
V3 / V2 = x1,69

Name of transformation: Text File Input - CSV + Calculator
Transformation description: Read a large CSV file, perform a single
calculation
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 100.000, runtime: 15,83s, speed: 6.318 rows/s
V3 results, rows: 100.000, runtime: 12,83s, speed: 7.795 rows/s
V3 / V2 = x1,23


ADD SEQUENCE

Name of transformation: AddSequence
Transformation description: Tests adding a single Long to an empty row.
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 1.000.000, runtime: 3,09s, speed: 323.206 rows/s
V3 results, rows: 1.000.000, runtime: 2,08s, speed: 481.232 rows/s
V3 / V2 = x1,49

Name of transformation: AddSequence to 5 fields
Transformation description: Tests adding a single Long to a row of 5 fields
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 1.000.000, runtime: 9,91s, speed: 100.949 rows/s
V3 results, rows: 1.000.000, runtime: 2,09s, speed: 477.555 rows/s
V3 / V2 = x4,73

Name of transformation: AddSequence to 10 fields
Transformation description: Tests adding a single Long to a row of 10 fields
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 1.000.000, runtime: 20,36s, speed: 49.118 rows/s
V3 results, rows: 1.000.000, runtime: 2,25s, speed: 444.444 rows/s
V3 / V2 = x9,05

Name of transformation: Add 2 sequences
Transformation description: Tests adding 2 sequences to an empty row.
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 1.000.000, runtime: 5,28s, speed: 189.322 rows/s
V3 results, rows: 1.000.000, runtime: 2,42s, speed: 412.882 rows/s
V3 / V2 = x2,18

Name of transformation: Add 2 sequence to 5 fields
Transformation description: Tests adding 2 sequences to a row of 5 fields
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 1.000.000, runtime: 14,09s, speed: 70.957 rows/s
V3 results, rows: 1.000.000, runtime: 2,86s, speed: 349.773 rows/s
V3 / V2 = x4,93

Name of transformation: Add 2 sequences to 10 fields
Transformation description: Tests adding 2 sequences to a row of 10 fields
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 1.000.000, runtime: 20,38s, speed: 49.080 rows/s
V3 results, rows: 1.000.000, runtime: 3,19s, speed: 313.676 rows/s
V3 / V2 = x6,39

Name of transformation: Add 4 sequences
Transformation description: Tests adding 4 sequences to an empty row.
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 1.000.000, runtime: 10,41s, speed: 96.089 rows/s
V3 results, rows: 1.000.000, runtime: 6,80s, speed: 147.124 rows/s
V3 / V2 = x1,53

Name of transformation: Add 4 sequence to 5 fields
Transformation description: Tests adding 4 sequences to a row of 5 fields
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 1.000.000, runtime: 18,92s, speed: 52.849 rows/s
V3 results, rows: 1.000.000, runtime: 8,81s, speed: 113.469 rows/s
V3 / V2 = x2,15

Name of transformation: Add 4 sequences to 10 fields
Transformation description: Tests adding 4 sequences to a row of 10 fields
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 1.000.000, runtime: 22,70s, speed: 44.045 rows/s
V3 results, rows: 1.000.000, runtime: 8,20s, speed: 121.907 rows/s
V3 / V2 = x2,77



ADD CONSTANT

Name of transformation: Constant
Transformation description: Tests the functionality and performance of the
"Constant" step.
----------------------------------------------------------------------------
-------------------------
V2 results, rows: 1.000.000, runtime: 5,91s, speed: 169.291 rows/s
V3 results, rows: 1.000.000, runtime: 0,98s, speed: 1.016.260 rows/s
V3 / V2 = x6,00










--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
-~----------~----~----~----~------~----~------~--~---