PDA

View Full Version : Stream lookup ...



t_christensen
02-16-2006, 05:59 AM
When can we expect "Stream lookup" to be able to compare fields with an comparator like "Database lookup" does ..?

And why was this not implemented from the beginning ..?
Any catch with the functionality ..?

MattCasters
02-16-2006, 06:18 AM
Hi Thomas,

>When can we expect "Stream lookup" to be able to compare fields with an comparator like "Database lookup" does ..?

Please keep an eye on trackers 1500 and 1501.
If I need to make a guess, in a month or two, maybe sooner.

>why was this not implemented from the beginning ..?

There can only be one answer to this question and many others alike: because it was not needed before. Don't get me wrong, I'm happy that Kettle is being used in a lot of roles like data migration, application integration, message parsing, etc. However, the principle role for which it was designed is data warehousing & ETL.
More on-topic, in this case, I used a hashtable to drive the lookups as this is the fasted way to look up values. It's as simple as that, no need to get worked up about it ;-)

>Any catch with the functionality ..?

Until tracker 1500 gets implemented, it runs in-memory. You might need to give Kettle more memory if you want to use it on large data sets.

Hope this clarifies things for you...

Best regards,
Matt