FYI: Kettle dependency management

[Jens Bleuel]

Just want let you know of this move of the jar dependencies and Ivy:

http://wiki.pentaho.com/display/EAI/...ncy+management

The status is in the "Work in Progress" section.

Feedback to the engineers is much appreciated.

Thanks & best regards / Vielen Dank und beste Gr

[Nicholas Goodman]

Kettle is unique amongst Pentaho projects in that
a) It gets significant, ongoing, real contributions externally and
b) Those contributions also include helper jars

re: a) Part of the reason, imho, that Kettle has been able to garner such participation is Matt's diligence on making sure the project is accessible to a huge range of part time developers. ie, SVN checkout as a project in eclipse, quick look over the straightforward Java, and you're able to do dev work. A move to Ivy (including req IvyDE for Eclipse devs, or other bootstrapping) does diminish this. I'm sure Pentaho feels pains on their builds which is why they've forced all their projects to Ivy for their own mgmt/convenience.

Just so I'm not painted as being negative: I believe that Pentahos benefit may be sufficient to warrant this increased complexity for casual Kettle devs.

re: b) Many of the maintainers and developers of steps / job entries bring libraries with them (which is why there are sooooo many jars). Currently, developers creating new steps can check their required libraries in along with their code. Does Pentaho plan to provide a way for Kettle developers to have "write" access to the Ivy repository (artifactory) for developers to update their dependency jars? Ie, for business to continue as usual a Kettle dev needs to be able to SVN commit (their code) and update their dependencies (artifactory).

Nick
PS - Writing a custom customer PDI plugin today, which is why this email popped back into my head.

On Nov 19, 2010, at 1:46 AM, Jens Bleuel wrote:
[color=blue]
> Just want let you know of this move of the jar dependencies and Ivy:
>
> http://wiki.pentaho.com/display/EAI/...ncy+management
>
> The status is in the "Work in Progress" section.
>
> Feedback to the engineers is much appreciated.
>
> Thanks & best regards / Vielen Dank und beste Gr

[Matt Casters]

Hi Nick,

You are obviously right on the money with your comments. However, the dev
team in Orlando promised me to
1) make Ivy Just Work(tm) with Kettle before the old Ant build file is
deprecated.
2) to make it as easy as possible to use and manage

When those things are in place I don't object to having Ivy on board at all
since I think that we all should be happy to see Kettle being used in other
projects. More and more, Kettle is becoming a platform against which
applications (BI Server, PRD, PME, Quipu, ...) are being written.
Because of this it becomes more and more important to be able to say which
jar files are required, which versions and so on. Obviously, this will also
tremendously help in speeding up the release cycles of Kettle.

Like I said, I'm convinced you are absolutely correct in saying that Ivy
will make adding new code to Kettle more complex. However, I'll do my
utmost best to help anyone with any problems because I believe that it can
increase adoption of Kettle when used as a platform or API.

As far as the dependencies of new steps, job entries and plugins is
concerned... Well, let me reply with a separate mail concerning this issue.
;-)

Matt

2010/11/27 Nicholas Goodman <ngoodman (AT) bayontechnologies (DOT) com>
[color=blue]
> Kettle is unique amongst Pentaho projects in that
> a) It gets significant, ongoing, real contributions externally and
> b) Those contributions also include helper jars
>
> re: a) Part of the reason, imho, that Kettle has been able to garner such
> participation is Matt's diligence on making sure the project is accessible
> to a huge range of part time developers. ie, SVN checkout as a project in
> eclipse, quick look over the straightforward Java, and you're able to do dev
> work. A move to Ivy (including req IvyDE for Eclipse devs, or other
> bootstrapping) does diminish this. I'm sure Pentaho feels pains on their
> builds which is why they've forced all their projects to Ivy for their own
> mgmt/convenience.
>
> Just so I'm not painted as being negative: I believe that Pentahos benefit
> may be sufficient to warrant this increased complexity for casual Kettle
> devs.
>
> re: b) Many of the maintainers and developers of steps / job entries bring
> libraries with them (which is why there are sooooo many jars). Currently,
> developers creating new steps can check their required libraries in along
> with their code. Does Pentaho plan to provide a way for Kettle developers
> to have "write" access to the Ivy repository (artifactory) for developers to
> update their dependency jars? Ie, for business to continue as usual a
> Kettle dev needs to be able to SVN commit (their code) and update their
> dependencies (artifactory).
>
> Nick
> PS - Writing a custom customer PDI plugin today, which is why this email
> popped back into my head.
>
> On Nov 19, 2010, at 1:46 AM, Jens Bleuel wrote:
>[color=green]
> > Just want let you know of this move of the jar dependencies and Ivy:
> >
> > http://wiki.pentaho.com/display/EAI/...ncy+management
> >
> > The status is in the "Work in Progress" section.
> >
> > Feedback to the engineers is much appreciated.
> >
> > Thanks & best regards / Vielen Dank und beste Gr

[Michael D'Amour]

Of all the things I have seen and heard about the proposed Kettle
migration to Ivy, one thing I have yet to hear is any feedback from
anyone actually giving the mdd-ivy branch a try.

You can pull the development branch of Kettle/Ivy at:
svn://source.pentaho.org/svnkettleroot/Kettle/branches/mdd-ivy

If you have Eclipse you can simply checkout the project and if you
have ivyde it will automatically resolve the dependencies which are
CLEARLY stated in the ivy.xml. If you do not have ivyde, you can just
run "ant resolve" and then either manually add the jars to your
project .classpath or you can run "ant create-dot-classpath" and have
this done for you. Just refresh the project (to pickup the .classpath
update) and Eclipse should compile the source just fine.

If you don't have Eclipse you should still call "ant resolve" and then
using your IDE, add the jars to the classpath. All "resolved"
resources are located in resolved-* folder names. I have added the
capability of resolving external plugins such that they may be
"installed" in the Kettle "ant distrib". This means that we can
pretty easily add third-party/community/external plugins to our build
process:

<dependency org="pentaho" name="pdi-hadoop-plugin" rev="$
{dependency.pdi-hadoop.revision}" conf="plugin->default"
changing="true" transitive="false">
<artifact name="pdi-hadoop-plugin" type="zip" />
</dependency>

The build.properties has the substitution for the revision (just like
all other ant properties). These plugins will be resolved in a
resolved-plugins folder which will be extracted by the distrib task.
We can easily create a new plugin outside of Kettle's core source tree
and incorporate it into Kettle's build by adding it to the plugin
dependencies! These plugins can be independently added and monitored
to our CI environment, with the capability of automatically triggering
a new kettle build if a plugin changes. This will allow us to know
early and often of the impact of any changes.

So what's my point here? Well, I'd like some feedback based on actual
use. Was it easier than expected? Were there problems? Was the
ivy.xml easy to understand?

It should be noted that I have *not* made very many changes to the
Kettle build process. You can still checkout Kettle and immediately
invoke "ant distrib".

-Mike

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.