PDA

View Full Version : Catching up with Kettle REMIX



MattCasters
11-12-2018, 04:50 AM
Dear Kettle and Neo4j friends,
Since I joined the Neo4j (https://neo4j.com/blog/matt-casters-behind-kettle-has-joined-neo4j/) team in April I haven’t given you any updates despite the fact that a lot of activity has been taking place in both the Neo4j and Kettle realms.
First and foremost, you can grab the cool Neo4j plugins from kettle.neo4j.be (http://kettle.neo4j.be) (the plugin in the marketplace is always out of date since it takes weeks to update the metadata).
Then based on valuable feedback from community members we’ve updated the DataSet plugin (https://github.com/mattcasters/pentaho-pdi-dataset) (including unit testing) to include relative paths for filenames (for easier git support), to avoid modifying transformation metadata and to set custom variables or parameters.
I’ve also created a plugin to debug (https://github.com/mattcasters/kettle-debug-plugin) transformations and jobs a bit easier. You can do things like set specific logging levels on steps (or only for a few rows) and work with zoom levels.
Then, back on the subject of Neo4j, I’ve created a plugin to log (https://github.com/mattcasters/kettle-neo4j-logging) the execution results of transformations and jobs (and a bit of their metadata) to Neo4j.
Those working with Azure might enjoy the Event Hubs plugins (https://github.com/mattcasters/kettle-azure-event-hubs) for a bit of data streaming action in Kettle.
The Kettle Needful Things plugin aims to fix bugs and solve silly problems in Kettle. For now it sets the correct local metastore on Carte servers AND… features a new launcher script called Maitre (https://github.com/mattcasters/kettle-needful-things/wiki/Maitre). Maitre supports transformations and jobs, local, remote and clustered execution.
The Kettle Environment plugin aims to take a stab at lifecycle management by allowing you to define a list of Environments:
http://www.ibridge.be/wp-content/uploads/2018/11/spoon-environments-dialog.png (http://www.ibridge.be/wp-content/uploads/2018/11/spoon-environments-dialog.png)
In each Environment you can set all sorts of metadata but also the location of the Kettle and MetaStore home folders.
http://www.ibridge.be/wp-content/uploads/2018/11/spoon-environment-dialog-1.png (http://www.ibridge.be/wp-content/uploads/2018/11/spoon-environment-dialog-1.png)
Finally, because downloading, patching, installing and configuring all this is a lot of work, I’ve created an automated process which does this for you on a daily bases (for testing) and so you can download Kettle Community Edition version 8.1.0.0 patched to 8.1.0.4 with all the extra plugins above in its 1GB glory at : remix.kettle.be (http://remix.kettle.be)
To get it on your server simply run:
wget remix.kettle.be -O remix.zipYou can also give these plugins (Except for Needful-things and Environment) on my sandbox WebSpoon server (http://sandbox.kettle.be:8080/spoon/spoon). You can easily run your own WebSpoon from the also daily updated docker container (https://hub.docker.com/r/mattcasters/webspoon-neo4j/).
If you have suggestions, bugs, rants, please feel free to leave them here or in the respective github projects. Any feedback is as always more than welcome. In fact, thanks you all for the feedback given so far. It’s making all the difference. If you feel the need to contribute more opinions on the subjects of Kettle feel free to send me a mail (mattcasters at gmail dot com) to join our kettle-community (https://kettle-community.slack.com) Slack channel.
Enjoy!
Matt


More... (http://www.ibridge.be/?p=323)