A post on the forum lead me to do some performance tests with the latest
error handling code that we added.
The idea is that if we can now determine on a row-bases which row wasn't
inserted into a table, we can perhaps get better performance by just letting
the errors fly.
It's a technique that certain ETL tools (the ones with tight integration
with the database) use to enhance performance, but there was no telling what
it would do on a streaming based tool like Kettle.

In attachement are the results as well as the 12 transformations to see what
was going on.

The conclusion for me is that the speed difference is such that we might
need to implement a new strategy inside of Insert/Update to speed it up.
An extra checkboxe should do the trick: [x] table has unique constraint on
the specified key(s)
In that case we know the DB is going to throw an error on insert.

In the mean time that this feature pops up, there is now a workaround...

This is the the DDL to create the table on MySQL, so you get an idea:

CREATE TABLE s_1m_customers
(
CUSTOMERNR INT
, NAME VARCHAR(26)
, FIRSTNAME VARCHAR(14)
, LANGUAGE VARCHAR(1)
, GENDER VARCHAR(8)
, STREET VARCHAR(16)
, HOUSNR VARCHAR(4)
, BUSNR VARCHAR(4)
, ZIPCODE VARCHAR(10)
, LOCATION VARCHAR(14)
, COUNTRY VARCHAR(21)
, DATE_OF_BIRTH DATETIME
, primary key(CUSTOMERNR)
)
;

All the best,

Matt
____________________________________________
Matt Casters, Chief Data Integration
Pentaho, Open Source Business Intelligence
http://www.pentaho.org <http://www.pentaho.org/> -- mcasters (AT) pentaho (DOT) org
Tel. +32 (0) 486 97 29 37



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
-~----------~----~----~----~------~----~------~--~---