Stream Schema Merge

The contribution

Our friends at Graphiq did it again; Not only they recently submitted a great contirbution for the Json Input plugin that we incorporated in 6.1, recently they added a new contribution, the Stream Schema Merge plugin.

Andrew did an amazing blog post describing it; I can't possibly do better than what they did, so I'm really just highlighting their effort. But I'd like to just briefly describe what it does

How it works

I admit at first I was a bit confused; Why the hell do I need a step to do something that PDI does natively?

Then I saw this:

And this:

Then it all made sense! It's bloody genius! The trick here is that with this step there's no longer the requirement of converting all streams to the same structure! Not only this simplifies the transformation a lot, it really improves performance: Every time we change the structure of a stream there's a cost of building a new Object[][].

And with this step we don't need that.

Amazing job guys! Honestly, after seeing this, the question I ask myself is why doesn't PDI just behave like this by default?? And on that, we may talk again, Andrew and team :p