Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: FYI: Kettle dependency management

  1. #1
    Jens Bleuel Guest

    Default FYI: Kettle dependency management

    Just want let you know of this move of the jar dependencies and Ivy:

    http://wiki.pentaho.com/display/EAI/...ncy+management

    The status is in the "Work in Progress" section.

    Feedback to the engineers is much appreciated.

    Thanks & best regards / Vielen Dank und beste Gr

  2. #2
    Nicholas Goodman Guest

    Default Re: FYI: Kettle dependency management

    Kettle is unique amongst Pentaho projects in that
    a) It gets significant, ongoing, real contributions externally and
    b) Those contributions also include helper jars

    re: a) Part of the reason, imho, that Kettle has been able to garner such participation is Matt's diligence on making sure the project is accessible to a huge range of part time developers. ie, SVN checkout as a project in eclipse, quick look over the straightforward Java, and you're able to do dev work. A move to Ivy (including req IvyDE for Eclipse devs, or other bootstrapping) does diminish this. I'm sure Pentaho feels pains on their builds which is why they've forced all their projects to Ivy for their own mgmt/convenience.

    Just so I'm not painted as being negative: I believe that Pentahos benefit may be sufficient to warrant this increased complexity for casual Kettle devs.

    re: b) Many of the maintainers and developers of steps / job entries bring libraries with them (which is why there are sooooo many jars). Currently, developers creating new steps can check their required libraries in along with their code. Does Pentaho plan to provide a way for Kettle developers to have "write" access to the Ivy repository (artifactory) for developers to update their dependency jars? Ie, for business to continue as usual a Kettle dev needs to be able to SVN commit (their code) and update their dependencies (artifactory).

    Nick
    PS - Writing a custom customer PDI plugin today, which is why this email popped back into my head.

    On Nov 19, 2010, at 1:46 AM, Jens Bleuel wrote:
    [color=blue]
    > Just want let you know of this move of the jar dependencies and Ivy:
    >
    > http://wiki.pentaho.com/display/EAI/...ncy+management
    >
    > The status is in the "Work in Progress" section.
    >
    > Feedback to the engineers is much appreciated.
    >
    > Thanks & best regards / Vielen Dank und beste Gr

  3. #3
    Matt Casters Guest

    Default Re: FYI: Kettle dependency management

    Hi Nick,

    You are obviously right on the money with your comments. However, the dev
    team in Orlando promised me to
    1) make Ivy Just Work(tm) with Kettle before the old Ant build file is
    deprecated.
    2) to make it as easy as possible to use and manage

    When those things are in place I don't object to having Ivy on board at all
    since I think that we all should be happy to see Kettle being used in other
    projects. More and more, Kettle is becoming a platform against which
    applications (BI Server, PRD, PME, Quipu, ...) are being written.
    Because of this it becomes more and more important to be able to say which
    jar files are required, which versions and so on. Obviously, this will also
    tremendously help in speeding up the release cycles of Kettle.

    Like I said, I'm convinced you are absolutely correct in saying that Ivy
    will make adding new code to Kettle more complex. However, I'll do my
    utmost best to help anyone with any problems because I believe that it can
    increase adoption of Kettle when used as a platform or API.

    As far as the dependencies of new steps, job entries and plugins is
    concerned... Well, let me reply with a separate mail concerning this issue.
    ;-)

    Matt

    2010/11/27 Nicholas Goodman <ngoodman (AT) bayontechnologies (DOT) com>
    [color=blue]
    > Kettle is unique amongst Pentaho projects in that
    > a) It gets significant, ongoing, real contributions externally and
    > b) Those contributions also include helper jars
    >
    > re: a) Part of the reason, imho, that Kettle has been able to garner such
    > participation is Matt's diligence on making sure the project is accessible
    > to a huge range of part time developers. ie, SVN checkout as a project in
    > eclipse, quick look over the straightforward Java, and you're able to do dev
    > work. A move to Ivy (including req IvyDE for Eclipse devs, or other
    > bootstrapping) does diminish this. I'm sure Pentaho feels pains on their
    > builds which is why they've forced all their projects to Ivy for their own
    > mgmt/convenience.
    >
    > Just so I'm not painted as being negative: I believe that Pentahos benefit
    > may be sufficient to warrant this increased complexity for casual Kettle
    > devs.
    >
    > re: b) Many of the maintainers and developers of steps / job entries bring
    > libraries with them (which is why there are sooooo many jars). Currently,
    > developers creating new steps can check their required libraries in along
    > with their code. Does Pentaho plan to provide a way for Kettle developers
    > to have "write" access to the Ivy repository (artifactory) for developers to
    > update their dependency jars? Ie, for business to continue as usual a
    > Kettle dev needs to be able to SVN commit (their code) and update their
    > dependencies (artifactory).
    >
    > Nick
    > PS - Writing a custom customer PDI plugin today, which is why this email
    > popped back into my head.
    >
    > On Nov 19, 2010, at 1:46 AM, Jens Bleuel wrote:
    >[color=green]
    > > Just want let you know of this move of the jar dependencies and Ivy:
    > >
    > > http://wiki.pentaho.com/display/EAI/...ncy+management
    > >
    > > The status is in the "Work in Progress" section.
    > >
    > > Feedback to the engineers is much appreciated.
    > >
    > > Thanks & best regards / Vielen Dank und beste Gr

  4. #4
    Michael D'Amour Guest

    Default Re: FYI: Kettle dependency management

    Of all the things I have seen and heard about the proposed Kettle
    migration to Ivy, one thing I have yet to hear is any feedback from
    anyone actually giving the mdd-ivy branch a try.

    You can pull the development branch of Kettle/Ivy at:
    svn://source.pentaho.org/svnkettleroot/Kettle/branches/mdd-ivy

    If you have Eclipse you can simply checkout the project and if you
    have ivyde it will automatically resolve the dependencies which are
    CLEARLY stated in the ivy.xml. If you do not have ivyde, you can just
    run "ant resolve" and then either manually add the jars to your
    project .classpath or you can run "ant create-dot-classpath" and have
    this done for you. Just refresh the project (to pickup the .classpath
    update) and Eclipse should compile the source just fine.

    If you don't have Eclipse you should still call "ant resolve" and then
    using your IDE, add the jars to the classpath. All "resolved"
    resources are located in resolved-* folder names. I have added the
    capability of resolving external plugins such that they may be
    "installed" in the Kettle "ant distrib". This means that we can
    pretty easily add third-party/community/external plugins to our build
    process:

    <dependency org="pentaho" name="pdi-hadoop-plugin" rev="$
    {dependency.pdi-hadoop.revision}" conf="plugin->default"
    changing="true" transitive="false">
    <artifact name="pdi-hadoop-plugin" type="zip" />
    </dependency>

    The build.properties has the substitution for the revision (just like
    all other ant properties). These plugins will be resolved in a
    resolved-plugins folder which will be extracted by the distrib task.
    We can easily create a new plugin outside of Kettle's core source tree
    and incorporate it into Kettle's build by adding it to the plugin
    dependencies! These plugins can be independently added and monitored
    to our CI environment, with the capability of automatically triggering
    a new kettle build if a plugin changes. This will allow us to know
    early and often of the impact of any changes.

    So what's my point here? Well, I'd like some feedback based on actual
    use. Was it easier than expected? Were there problems? Was the
    ivy.xml easy to understand?

    It should be noted that I have *not* made very many changes to the
    Kettle build process. You can still checkout Kettle and immediately
    invoke "ant distrib".

    -Mike

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.