Hitachi Vantara Pentaho Community Forums
Results 1 to 27 of 27

Thread: Plugins and a market place

  1. #1
    Matt Casters Guest

    Default Plugins and a market place

    Dear Kettle devs,

    For a while now we've been thinking about creating a market place for
    Pentaho related plugins.
    The constant influx of new steps, job entries and Spoon plugins makes it
    increasingly hard to do proper dependency management (see previous mail).
    One solution could be to push away more and more of these steps and job
    entries to external plugins. However, we fear that developers might see
    those as second class citizens compared to "core" components of
    Kettle. Developers might think that for the following reasons:
    1) Plugins are listed on a wiki page but otherwise not structurally
    described.
    2) Installation of the plugins is manual process and technical of nature,
    you have to take into account versions, and so on.
    3) Communication methods with the plugin authors (e-mail, JIRA, forum) are
    not always clear. (usually not!)
    4) The plugins are not maintained (code fixes, translation/i18n, code
    migration, ...) the same way as the Kettle core codebase.

    I strongly think that having a market in place that would do away with all
    these problems would in fact promote the creation of new plugins even more.

    In short, Pentaho is looking for ways to make it easier to integrate new
    functionality in Kettle without having to resort to the monolithic structure
    we have in place now. Any ideas you might have in this respect are most
    welcome.

    Regards,

    Matt
    --
    Matt Casters <mcasters (AT) pentaho (DOT) org>
    Chief Data Integration, Kettle founder, Author of Pentaho Kettle
    Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
    (Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
    Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    Pentaho : The Commercial Open Source Alternative for Business Intelligence

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  2. #2
    Roland Bouman Guest

    Default Re: Plugins and a market place

    Hi All!

    Overall, a good idea.
    I do think that to make it work, there'd have to be clear admission
    criteria, and an independent board / comittee to judge submissions
    against these criteria.
    Feedback to devs should also be in terms of those criteria in case
    their submission is currently not admitted.

    One of the criteria I would suggest is availability of freely
    available documentation (so it can be linked to from the marketplace
    site).
    I would also suggest criteria for the content and structure of the
    documentation.
    The value in this is to make it easy for potential users to discover
    whether it makes sense for them to use a a particular plugin.

    kind regards,

    Roland

    On Mon, Nov 29, 2010 at 1:45 PM, Matt Casters <mcasters (AT) pentaho (DOT) org> wrote:[color=blue]
    > Dear Kettle devs,
    > For a while now we've been thinking about creating a market place for
    > Pentaho related plugins.
    > The constant influx of new steps, job entries and Spoon plugins makes it
    > increasingly hard to do proper dependency management (see previous mail).
    > One solution could be to push away more and more of these steps and job
    > entries to external plugins.

  3. #3
    Matt Casters Guest

    Default Re: Plugins and a market place

    Hi Roland,

    I'm not sure I would want to be restrictive at all. I want people to have
    as much fun with the plugins as possible. I *want* to have experimental,
    undocumented stuff in the market. I want that since I strongly believe
    that's how open source comes to the best results. However, I also would
    expect to have a system in place that would allow me to vote plugins up or
    down from within Spoon, integrated problem reporting &JIRA, mini-forums for
    feedback, and so on.

    Having integrated documentation however is a good point. Giving a plugin a
    dedicated wiki page to drop the docs into is a good idea. Forcing people to
    document is IMO over the top. Perhaps we should create a "Vote down: no
    documentation" option somewhere.

    All in all, if someone contributes a plugin to calculate Pi in arbitrary
    position to 1M decimals, we can safely say that this is something we would
    like to keep as a plugin because the use-case is a bit scientific/specific.
    However, for a lot of other plugins that are imaginable in the data
    generation, data quality, data preparation areas, that's not clear at all.
    Myself nor the folks at Pentaho, nor any committee or board I can think of
    would want to make that decision either.

    Personally I kinda like the Android market idea where folks can rate and
    review apps. Perhaps we can combine that open "anyone can contribute" idea
    with a lot of infrastructure to facilitate two-way-communications,
    development, translation, documentation and source code access.

    Just my two Euro-cents,
    Matt

    2010/11/29 Roland Bouman <roland.bouman (AT) gmail (DOT) com>

    > Hi All!
    >
    > Overall, a good idea.
    > I do think that to make it work, there'd have to be clear admission
    > criteria, and an independent board / comittee to judge submissions
    > against these criteria.
    > Feedback to devs should also be in terms of those criteria in case
    > their submission is currently not admitted.
    >
    > One of the criteria I would suggest is availability of freely
    > available documentation (so it can be linked to from the marketplace
    > site).
    > I would also suggest criteria for the content and structure of the
    > documentation.
    > The value in this is to make it easy for potential users to discover
    > whether it makes sense for them to use a a particular plugin.
    >
    > kind regards,
    >
    > Roland
    >
    > On Mon, Nov 29, 2010 at 1:45 PM, Matt Casters <mcasters (AT) pentaho (DOT) org>
    > wrote:
    > > Dear Kettle devs,
    > > For a while now we've been thinking about creating a market place for
    > > Pentaho related plugins.
    > > The constant influx of new steps, job entries and Spoon plugins makes it
    > > increasingly hard to do proper dependency management (see previous mail).
    > > One solution could be to push away more and more of these steps and job
    > > entries to external plugins. However, we fear that developers might see
    > > those as second class citizens compared to "core" components of
    > > Kettle. Developers might think that for the following reasons:
    > > 1) Plugins are listed on a wiki page but otherwise not structurally
    > > described.
    > > 2) Installation of the plugins is manual process and technical of nature,
    > > you have to take into account versions, and so on.
    > > 3) Communication methods with the plugin authors (e-mail, JIRA, forum)

    > are
    > > not always clear. (usually not!)
    > > 4) The plugins are not maintained (code fixes, translation/i18n, code
    > > migration, ...) the same way as the Kettle core codebase.
    > > I strongly think that having a market in place that would do away with

    > all
    > > these problems would in fact promote the creation of new plugins even

    > more.
    > > In short, Pentaho is looking for ways to make it easier to integrate new
    > > functionality in Kettle without having to resort to

    > the monolithic structure
    > > we have in place now. Any ideas you might have in this respect are most
    > > welcome.
    > > Regards,
    > >
    > > Matt
    > > --
    > > Matt Casters <mcasters (AT) pentaho (DOT) org>
    > > Chief Data Integration, Kettle founder, Author of Pentaho Kettle
    > > Solutions (Wiley)
    > > Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    > > Pentaho : The Commercial Open Source Alternative for Business

    > Intelligence
    > >
    > > --
    > > You received this message because you are subscribed to the Google Groups
    > > "kettle-developers" group.
    > > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > > To unsubscribe from this group, send email to
    > > kettle-developers+unsubscribe (AT) g...oups (DOT) com>

    > .
    > > For more options, visit this group at
    > > http://groups.google.com/group/kettle-developers?hl=en.
    > >

    >
    >
    >
    > --
    > Roland Bouman
    > blog: http://rpbouman.blogspot.com/
    > twitter: @rolandbouman
    >
    > Author of "Pentaho Solutions: Business Intelligence and Data
    > Warehousing with Pentaho and MySQL",
    > http://tinyurl.com/lvxa88 (Wiley, ISBN: 978-0-470-48432-6)
    >
    > Author of "Pentaho Kettle Solutions: Building Open Source ETL
    > Solutions with Pentaho Data Integration",
    > http://tinyurl.com/33r7a8m (Wiley, ISBN: 978-0-470-63517-9)
    >
    > --
    > You received this message because you are subscribed to the Google Groups
    > "kettle-developers" group.
    > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > To unsubscribe from this group, send email to
    > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    > .
    > For more options, visit this group at
    > http://groups.google.com/group/kettle-developers?hl=en.
    >
    >



    --
    Matt Casters <mcasters (AT) pentaho (DOT) org>
    Chief Data Integration, Kettle founder, Author of Pentaho Kettle
    Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
    (Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
    Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    Pentaho : The Commercial Open Source Alternative for Business Intelligence

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  4. #4
    Roland Bouman Guest

    Default Re: Plugins and a market place

    Hi Matt, all,

    you raise a number of good points I hadn't thought about.

    My reply mostly reflects my own experiences hunting for plugins.

    With a few exceptions (web browser plugins), whether its CMS-related
    (drupal), web development (jQuery), I find I spend oodles of time
    downloading and trying plugins just to find out what they actually do.
    In many cases, documentation is so sparse that it's hard to even
    comprehend what the function of the plugin is. Just a little bit of
    documentation (screenshot + typical example) would remedey say 50% of
    those problems.

    But indeed maybe documentation criteria is a bit over the top. The
    voting system may indeed work as a filter - thanks for pointing that
    out.

    regards,

    Roland

    On Mon, Nov 29, 2010 at 2:05 PM, Matt Casters <mcasters (AT) pentaho (DOT) org> wrote:[color=blue]
    > Hi Roland,
    > I'm not sure I would want to be restrictive at all.

  5. #5
    Brandon Jackson Guest

    Default Re: Plugins and a market place

    You'll have to catch the plugin at the source where developers feed for
    information. That could be a dev list registration like this, or the
    pentaho forums. If there was a simple mechanism to 'request' a GUID for
    use in a plugin, then you could get an idea of how many are being attempted.
    The same GUID combined with a plugin.xml could give some catalog of
    plugins. The XML file could have a URLs inside that show a preferred link
    where 'loved' plugins could reside with a pentaho controlled URL, other URLs
    like 'author home' could point to at least another source.

    If each plugin had a GUID, then maybe an option in spoon would be to scan
    and report which GUIDs are used. A context menu on a step or job and allow
    voting up or down on versions of plugins, much like iTunes.

    It would probably be cool if there were a utility that could scan all the
    jobs and transformations just collecting the GUIDs and occurrence counts.
    (Popularity). You would especially want to make this option available on
    first run of a new version. (May reveal cracks before first run of a new
    version.)


    On Mon, Nov 29, 2010 at 7:23 AM, Roland Bouman <roland.bouman (AT) gmail (DOT) com>wrote:

    > Hi Matt, all,
    >
    > you raise a number of good points I hadn't thought about.
    >
    > My reply mostly reflects my own experiences hunting for plugins.
    >
    > With a few exceptions (web browser plugins), whether its CMS-related
    > (drupal), web development (jQuery), I find I spend oodles of time
    > downloading and trying plugins just to find out what they actually do.
    > In many cases, documentation is so sparse that it's hard to even
    > comprehend what the function of the plugin is. Just a little bit of
    > documentation (screenshot + typical example) would remedey say 50% of
    > those problems.
    >
    > But indeed maybe documentation criteria is a bit over the top. The
    > voting system may indeed work as a filter - thanks for pointing that
    > out.
    >
    > regards,
    >
    > Roland
    >
    > On Mon, Nov 29, 2010 at 2:05 PM, Matt Casters <mcasters (AT) pentaho (DOT) org>
    > wrote:
    > > Hi Roland,
    > > I'm not sure I would want to be restrictive at all. I want people to

    > have
    > > as much fun with the plugins as possible. I want to have experimental,
    > > undocumented stuff in the market. I want that since I strongly believe
    > > that's how open source comes to the best results. However, I also would
    > > expect to have a system in place that would allow me to vote plugins up

    > or
    > > down from within Spoon, integrated problem reporting &JIRA, mini-forums

    > for
    > > feedback, and so on.
    > > Having integrated documentation however is a good point. Giving a plugin

    > a
    > > dedicated wiki page to drop the docs into is a good idea. Forcing people

    > to
    > > document is IMO over the top. Perhaps we should create a "Vote down: no
    > > documentation" option somewhere.
    > > All in all, if someone contributes a plugin to calculate Pi in arbitrary
    > > position to 1M decimals, we can safely say that this is something we

    > would
    > > like to keep as a plugin because the use-case is a bit

    > scientific/specific.
    > > However, for a lot of other plugins that are imaginable in the data
    > > generation, data quality, data preparation areas, that's not clear at

    > all.
    > > Myself nor the folks at Pentaho, nor any committee or board I can think

    > of
    > > would want to make that decision either.
    > > Personally I kinda like the Android market idea where folks can rate and
    > > review apps. Perhaps we can combine that open "anyone can contribute"

    > idea
    > > with a lot of infrastructure to facilitate two-way-communications,
    > > development, translation, documentation and source code access.
    > > Just my two Euro-cents,
    > > Matt
    > >
    > > 2010/11/29 Roland Bouman <roland.bouman (AT) gmail (DOT) com>
    > >>
    > >> Hi All!
    > >>
    > >> Overall, a good idea.
    > >> I do think that to make it work, there'd have to be clear admission
    > >> criteria, and an independent board / comittee to judge submissions
    > >> against these criteria.
    > >> Feedback to devs should also be in terms of those criteria in case
    > >> their submission is currently not admitted.
    > >>
    > >> One of the criteria I would suggest is availability of freely
    > >> available documentation (so it can be linked to from the marketplace
    > >> site).
    > >> I would also suggest criteria for the content and structure of the
    > >> documentation.
    > >> The value in this is to make it easy for potential users to discover
    > >> whether it makes sense for them to use a a particular plugin.
    > >>
    > >> kind regards,
    > >>
    > >> Roland
    > >>
    > >> On Mon, Nov 29, 2010 at 1:45 PM, Matt Casters <mcasters (AT) pentaho (DOT) org>
    > >> wrote:
    > >> > Dear Kettle devs,
    > >> > For a while now we've been thinking about creating a market place for
    > >> > Pentaho related plugins.
    > >> > The constant influx of new steps, job entries and Spoon plugins makes

    > it
    > >> > increasingly hard to do proper dependency management (see previous
    > >> > mail).
    > >> > One solution could be to push away more and more of these steps and

    > job
    > >> > entries to external plugins. However, we fear that developers might

    > see
    > >> > those as second class citizens compared to "core" components of
    > >> > Kettle. Developers might think that for the following reasons:
    > >> > 1) Plugins are listed on a wiki page but otherwise not structurally
    > >> > described.
    > >> > 2) Installation of the plugins is manual process and technical of
    > >> > nature,
    > >> > you have to take into account versions, and so on.
    > >> > 3) Communication methods with the plugin authors (e-mail, JIRA, forum)
    > >> > are
    > >> > not always clear. (usually not!)
    > >> > 4) The plugins are not maintained (code fixes, translation/i18n, code
    > >> > migration, ...) the same way as the Kettle core codebase.
    > >> > I strongly think that having a market in place that would do away with
    > >> > all
    > >> > these problems would in fact promote the creation of new plugins even
    > >> > more.
    > >> > In short, Pentaho is looking for ways to make it easier to integrate

    > new
    > >> > functionality in Kettle without having to resort to
    > >> > the monolithic structure
    > >> > we have in place now. Any ideas you might have in this respect are

    > most
    > >> > welcome.
    > >> > Regards,
    > >> >
    > >> > Matt
    > >> > --
    > >> > Matt Casters <mcasters (AT) pentaho (DOT) org>
    > >> > Chief Data Integration, Kettle founder, Author of Pentaho Kettle
    > >> > Solutions (Wiley)
    > >> > Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    > >> > Pentaho : The Commercial Open Source Alternative for Business
    > >> > Intelligence
    > >> >
    > >> > --
    > >> > You received this message because you are subscribed to the Google
    > >> > Groups
    > >> > "kettle-developers" group.
    > >> > To post to this group, send email to

    > kettle-developers (AT) googlegroups (DOT) com.
    > >> > To unsubscribe from this group, send email to
    > >> > kettle-developers+unsubscribe (AT) g...oups (DOT) com>

    > .
    > >> > For more options, visit this group at
    > >> > http://groups.google.com/group/kettle-developers?hl=en.
    > >> >
    > >>
    > >>
    > >>
    > >> --
    > >> Roland Bouman
    > >> blog: http://rpbouman.blogspot.com/
    > >> twitter: @rolandbouman
    > >>
    > >> Author of "Pentaho Solutions: Business Intelligence and Data
    > >> Warehousing with Pentaho and MySQL",
    > >> http://tinyurl.com/lvxa88 (Wiley, ISBN: 978-0-470-48432-6)
    > >>
    > >> Author of "Pentaho Kettle Solutions: Building Open Source ETL
    > >> Solutions with Pentaho Data Integration",
    > >> http://tinyurl.com/33r7a8m (Wiley, ISBN: 978-0-470-63517-9)
    > >>
    > >> --
    > >> You received this message because you are subscribed to the Google

    > Groups
    > >> "kettle-developers" group.
    > >> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com

    > .
    > >> To unsubscribe from this group, send email to
    > >> kettle-developers+unsubscribe (AT) g...oups (DOT) com>

    > .
    > >> For more options, visit this group at
    > >> http://groups.google.com/group/kettle-developers?hl=en.
    > >>

    > >
    > >
    > >
    > > --
    > > Matt Casters <mcasters (AT) pentaho (DOT) org>
    > > Chief Data Integration, Kettle founder, Author of Pentaho Kettle
    > > Solutions (Wiley)
    > > Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    > > Pentaho : The Commercial Open Source Alternative for Business

    > Intelligence
    > >
    > > --
    > > You received this message because you are subscribed to the Google Groups
    > > "kettle-developers" group.
    > > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > > To unsubscribe from this group, send email to
    > > kettle-developers+unsubscribe (AT) g...oups (DOT) com>

    > .
    > > For more options, visit this group at
    > > http://groups.google.com/group/kettle-developers?hl=en.
    > >

    >
    >
    >
    > --
    > Roland Bouman
    > blog: http://rpbouman.blogspot.com/
    > twitter: @rolandbouman
    >
    > Author of "Pentaho Solutions: Business Intelligence and Data
    > Warehousing with Pentaho and MySQL",
    > http://tinyurl.com/lvxa88 (Wiley, ISBN: 978-0-470-48432-6)
    >
    > Author of "Pentaho Kettle Solutions: Building Open Source ETL
    > Solutions with Pentaho Data Integration",
    > http://tinyurl.com/33r7a8m (Wiley, ISBN: 978-0-470-63517-9)
    >
    > --
    > You received this message because you are subscribed to the Google Groups
    > "kettle-developers" group.
    > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > To unsubscribe from this group, send email to
    > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    > .
    > For more options, visit this group at
    > http://groups.google.com/group/kettle-developers?hl=en.
    >
    >


    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  6. #6
    Jon Roberts Guest

    Default Re: Plugins and a market place

    While a plugin marketplace could help with some of the dependency issues
    with better documentation, I suspect that the issue will persist.

    A quick search reveals what you might expect. Dependency problems are not
    unique to kettle.
    http://www.google.com/search?rlz=1C1...age+dependency


    <http://www.google.com/search?rlz=1C1GPCK_enUS342US342&sourceid=chrome&ie=UTF-8&q=eclipse+package+dependency>Some
    type of packaging, central repositories, and associated management tools are
    usually employed to handle this. There are lots of examples for both OS
    (rpm, dpk) and programming environments (CPAN, pip - eggs - pypm). Given
    the widespread adoption of java and eclipse both, I'm somewhat surprised
    that a standard has not already emerged for this? Maybe there is one and
    I'm just unaware of it?

    On Mon, Nov 29, 2010 at 6:45 AM, Matt Casters <mcasters (AT) pentaho (DOT) org> wrote:

    > Dear Kettle devs,
    >
    > For a while now we've been thinking about creating a market place for
    > Pentaho related plugins.
    > The constant influx of new steps, job entries and Spoon plugins makes it
    > increasingly hard to do proper dependency management (see previous mail).
    > One solution could be to push away more and more of these steps and job
    > entries to external plugins. However, we fear that developers might see
    > those as second class citizens compared to "core" components of
    > Kettle. Developers might think that for the following reasons:
    > 1) Plugins are listed on a wiki page but otherwise not structurally
    > described.
    > 2) Installation of the plugins is manual process and technical of nature,
    > you have to take into account versions, and so on.
    > 3) Communication methods with the plugin authors (e-mail, JIRA, forum) are
    > not always clear. (usually not!)
    > 4) The plugins are not maintained (code fixes, translation/i18n, code
    > migration, ...) the same way as the Kettle core codebase.
    >
    > I strongly think that having a market in place that would do away with all
    > these problems would in fact promote the creation of new plugins even more.
    >
    > In short, Pentaho is looking for ways to make it easier to integrate new
    > functionality in Kettle without having to resort to the monolithic structure
    > we have in place now. Any ideas you might have in this respect are most
    > welcome.
    >
    > Regards,
    >
    > Matt
    > --
    > Matt Casters <mcasters (AT) pentaho (DOT) org>
    > Chief Data Integration, Kettle founder, Author of Pentaho Kettle Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
    > (Wiley<http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>
    > )
    > Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    > Pentaho : The Commercial Open Source Alternative for Business Intelligence
    >
    >
    > --
    > You received this message because you are subscribed to the Google Groups
    > "kettle-developers" group.
    > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > To unsubscribe from this group, send email to
    > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    > .
    > For more options, visit this group at
    > http://groups.google.com/group/kettle-developers?hl=en.
    >




    --
    Sincerely,

    Jon Roberts

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  7. #7
    Matt Casters Guest

    Default Re: Plugins and a market place

    Actually, Kettle plugins are loaded by separate class loaders. As such, a
    solid plugin system like a market would indeed lower dependencies since you
    don't have to ship with libfoo-2.1.3.jar in the main Kettle distribution.
    The jar file gets isolated and is loaded for a single specific plugin.

    We don't have to overstate the dependency issues we have now either.
    Usually these are minor things like upgrading a commons library from 3.1.2
    to 3.1.3 or going from POI 3.6 to 3.7. The problem is mainly located in the
    area of *knowing* which libraries are in conflict with other projects, not
    the resolution as such. For example, you want to have an answer to
    questions like: I want to get data from a Kettle transformation in report
    designer. Which libraries do I need, which of those do I already have in
    PRD and which are in conflict?

    Take care,

    Matt

    2010/11/29 Jon Roberts <bigjonroberts (AT) gmail (DOT) com>

    > While a plugin marketplace could help with some of the dependency issues
    > with better documentation, I suspect that the issue will persist.
    >
    > A quick search reveals what you might expect. Dependency problems are not
    > unique to kettle.
    >
    > http://www.google.com/search?rlz=1C1...age+dependency
    >
    >
    >
    > <http://www.google.com/search?rlz=1C1GPCK_enUS342US342&sourceid=chrome&ie=UTF-8&q=eclipse+package+dependency>Some
    > type of packaging, central repositories, and associated management tools are
    > usually employed to handle this. There are lots of examples for both OS
    > (rpm, dpk) and programming environments (CPAN, pip - eggs - pypm). Given
    > the widespread adoption of java and eclipse both, I'm somewhat surprised
    > that a standard has not already emerged for this? Maybe there is one and
    > I'm just unaware of it?
    >
    > On Mon, Nov 29, 2010 at 6:45 AM, Matt Casters <mcasters (AT) pentaho (DOT) org>wrote:
    >
    >> Dear Kettle devs,
    >>
    >> For a while now we've been thinking about creating a market place for
    >> Pentaho related plugins.
    >> The constant influx of new steps, job entries and Spoon plugins makes it
    >> increasingly hard to do proper dependency management (see previous mail).
    >> One solution could be to push away more and more of these steps and job
    >> entries to external plugins. However, we fear that developers might see
    >> those as second class citizens compared to "core" components of
    >> Kettle. Developers might think that for the following reasons:
    >> 1) Plugins are listed on a wiki page but otherwise not structurally
    >> described.
    >> 2) Installation of the plugins is manual process and technical of nature,
    >> you have to take into account versions, and so on.
    >> 3) Communication methods with the plugin authors (e-mail, JIRA, forum) are
    >> not always clear. (usually not!)
    >> 4) The plugins are not maintained (code fixes, translation/i18n, code
    >> migration, ...) the same way as the Kettle core codebase.
    >>
    >> I strongly think that having a market in place that would do away with all
    >> these problems would in fact promote the creation of new plugins even more.
    >>
    >> In short, Pentaho is looking for ways to make it easier to integrate new
    >> functionality in Kettle without having to resort to the monolithic structure
    >> we have in place now. Any ideas you might have in this respect are most
    >> welcome.
    >>
    >> Regards,
    >>
    >> Matt
    >> --
    >> Matt Casters <mcasters (AT) pentaho (DOT) org>
    >> Chief Data Integration, Kettle founder, Author of Pentaho Kettle
    >> Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
    >> (Wiley<http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>
    >> )
    >> Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    >> Pentaho : The Commercial Open Source Alternative for Business Intelligence
    >>
    >>
    >> --
    >> You received this message because you are subscribed to the Google Groups
    >> "kettle-developers" group.
    >> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    >> To unsubscribe from this group, send email to
    >> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    >> .
    >> For more options, visit this group at
    >> http://groups.google.com/group/kettle-developers?hl=en.
    >>

    >
    >
    >
    > --
    > Sincerely,
    >
    > Jon Roberts
    >
    > --
    > You received this message because you are subscribed to the Google Groups
    > "kettle-developers" group.
    > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > To unsubscribe from this group, send email to
    > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    > .
    > For more options, visit this group at
    > http://groups.google.com/group/kettle-developers?hl=en.
    >




    --
    Matt Casters <mcasters (AT) pentaho (DOT) org>
    Chief Data Integration, Kettle founder, Author of Pentaho Kettle
    Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
    (Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
    Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    Pentaho : The Commercial Open Source Alternative for Business Intelligence

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  8. #8
    Tony Cook Guest

    Default Re: Plugins and a market place

    Now this is a good idea! The opportunities for PDI as a

  9. #9
    Mark Hall Guest

    Default Re: Plugins and a market place

    The Weka project went through just this exercise with the last release
    of the development version in late July. I introduced a new package
    management system, based heavily on what the R statistical software
    uses. All contributions were split out of the core of Weka along with
    many core algorithms and made into packages. The whole shebang is
    hosted on Sourceforge - this includes a central package meta data
    repository and the actual packages themselves. Weka allows for
    dependencies between not only the base system but other packages too
    (something that Kettle doesn't concern itself with given its separate
    class loader per plugin approach) and the package management system
    takes care of handling dependencies and detecting conflicts. Users can
    use either a command line or GUI client to download and install
    packages.

    We have a concept of "official" and "unofficial" packages for Weka.
    Official packages get their meta data hosted at the central repository
    (the package itself can reside anywhere that is http accessible) and
    are verified by the Weka team (to the extent that the package
    functions and does not contain malware etc.). Unofficial packages
    don't get their meta data hosted at the central repository and can be
    installed by just pointing the package manager at their URL. The
    unofficial route allows folks to quickly make (possibly) experimental
    stuff available without having to go through the official channel.

    The main motivation for the Weka project was to reduce the burden of
    supporting external contributions. It makes a clear delineation
    between what is provided by the core Weka team and what is a
    contribution. A secondary goal was to open up Weka to the use of
    third-party libraries (something that has been avoided like the plague
    in the past :-)). Third-party library related issues/problems are now
    clearly the domain of package contributers who include them in their
    packages.

    This has all been working just swimmingly since the release. Packages
    have been downloaded via the package manager more than 33,000 times
    since late July.

    The package manager code is independent of Weka and is checked in to
    Pentaho's public SVN.

    For those who are interested, the central repository is at:
    http://weka.sourceforge.net/packageMetaData/

    Information on the clients and package structure is at:
    http://weka.wikispaces.com/How+do+I+...age+manager%3F
    http://weka.wikispaces.com/How+are+p...ment+system%3F

    Cheers,
    Mark.


    On Tue, Nov 30, 2010 at 1:51 AM, Roland Bouman <roland.bouman (AT) gmail (DOT) com> wrote:[color=blue]
    > Hi All!
    >
    > Overall, a good idea.
    > I do think that to make it work, there'd have to be clear admission
    > criteria, and an independent board / comittee to judge submissions
    > against these criteria.
    > Feedback to devs should also be in terms of those criteria in case
    > their submission is currently not admitted.
    >
    > One of the criteria I would suggest is availability of freely
    > available documentation (so it can be linked to from the marketplace
    > site).
    > I would also suggest criteria for the content and structure of the
    > documentation.
    > The value in this is to make it easy for potential users to discover
    > whether it makes sense for them to use a a particular plugin.
    >
    > kind regards,
    >
    > Roland
    >
    > On Mon, Nov 29, 2010 at 1:45 PM, Matt Casters <mcasters (AT) pentaho (DOT) org> wrote:[color=green]
    >> Dear Kettle devs,
    >> For a while now we've been thinking about creating a market place for
    >> Pentaho related plugins.
    >> The constant influx of new steps, job entries and Spoon plugins makes it
    >> increasingly hard to do proper dependency management (see previous mail)..
    >> One solution could be to push away more and more of these steps and job
    >> entries to external plugins.

  10. #10
    codeoncoffee Guest

    Default Re: Plugins and a market place

    Actually there is a standard for this. It's the underpinning of the
    Eclipse plugin system, OSGI:
    http://en.wikipedia.org/wiki/OSGi
    http://www.osgi.org/About/WhyOSGi

    The Eclipse project migrated to OSGI in v3 and maintains what's
    considered the reference implementation in Equinox ( http://www.eclipse.org/equinox/
    ). Recently many other companies and projects have migrated to OSGI
    themselves or provided support for it including: JBoss, WebShere,
    GlassFish, Netbeans, Weblogic, SpringSource, Apache Sling... and the
    list goes on.

    OSGI provides a component architecture framework for applications
    based on plain Jars with runtime dependency information contained as
    metadata in the jar manifests. The OSGI runtime manages wiring up the
    component's (Bundles) with needed library classes and provides several
    ways for services to be shared and tracked between bundles.

    There's even built-in support for downloading dependencies over the
    internet via a Bundle Repository (OBR) such as SpringSource's (
    http://ebr.springsource.com/repository/app/bundle ) or though a Maven
    repository. If you've installed a plugin in Eclipse you've seen OSGI
    in action. You can install/start/stop/restart a plugin without
    bringing down the application.

    There's quite a large OSGI community out there with a great number of
    bundles to take advantage of and resources for developers. And they're
    not light-weights. Last I checked there are around 10 JSRs based on
    OSGI. All of the major IDEs support development of Bundles and there
    excellent plugins for Ant and Maven to support building. But as I
    mentioned, they're just Jars. It doesn't take much to get started.

    If you can't tell I'm a big proponent . Pentaho has large number of
    plugin/service/configuration systems, all different and incomplete. As
    we further integrate our products we're duplicating a lot of
    integration code (plugin wiring, initialization, configuration) and
    maintaining and extending the various systems in parallel. All that
    could be standardized under a unified component (plugin) system.

    For the past couple months I've been leading a group here
    investigating just this thing. We decided Kettle would be the natural
    place to investigate an OSGI migration. It's plugin system is quite
    similar in many ways and it has a very rich component set with many
    plugins. Also it touches just about all of our technologies and
    projects I just recently checked-in the first code:

    OSGI-ified version of Kettle with preliminary support for Bundle
    provided steps checked-in at:
    (svn://source.pentaho.org/svnkettleroot/Kettle/branches/osgi)

    This past week I worked on converting over the MaxMind Lookup step.
    It's also checked-in:
    (svn://source.pentaho.org/svnkettleroot/plugins/MaxMindGeoIPLookup/
    branches/osgi)

    This effort is nowhere near complete and is provided as a test-bed for
    the technology. There is wiki page up outlining the new OSGI-based
    plugin system and I am looking for feedback!:
    http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle

    While this is all preliminary, I'm scheduled to host a brown-bag for
    the Orlando developers sharing what I've learning. Doug Moran has also
    asked me to present this formally to the Kettle Devs in an upcoming
    community WebEx as a potential way forward.

    Cheers,
    Nick Baker



    On Nov 29, 9:48

  11. #11
    Nicholas Goodman Guest

    Default Re: Plugins and a market place

    On Nov 29, 2010, at 4:57 PM, codeoncoffee wrote:

    > This effort is nowhere near complete and is provided as a test-bed for
    > the technology. There is wiki page up outlining the new OSGI-based
    > plugin system and I am looking for feedback!:
    > http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle


    OSGi is a great choice. Hazaa!

    Nick

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  12. #12
    dmoran Guest

    Default Re: Plugins and a market place

    For the popularity issue I think the more we can automate the better.
    We could add a heartbeat that reports stats back and is available with
    the download that shows how many times it has been installed,
    uninstalled and run - that would also help with the niche type plugins
    where total downloads is small, but people who do use it - use it a
    lot. The key to success in something like this is automation and low
    touch.

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  13. #13
    Tony Cook Guest

    Default Re: Plugins and a market place

    Reading through the OSGI stuff and looking at the sample code, even though I
    don't fully understand it, it looks well thought through and I can sense the
    enthusiasm radiating from Orlando!

    I am just wondering if PDI really needs this ability to install modules
    dynamically without a restart. Currently to install a plugin you just have
    to copy the files (and any required jars) into the relevant plugin folder
    and then restart Spoon. The benefit of a marketplace is to aid "discovery"
    by users, but I'm sure a user would be more than happy to restart after
    installing a plugin (like with Firefox).

    Unless there is a use-case of PDI that requires it to be permanently
    running? Would this use case need to install/uninstall plugins at run time
    with no possibility of a restart?

    Tony


    On Tue, Nov 30, 2010 at 1:52 AM, Nicholas Goodman <
    ngoodman (AT) bayontechnologies (DOT) com> wrote:

    > On Nov 29, 2010, at 4:57 PM, codeoncoffee wrote:
    >
    > This effort is nowhere near complete and is provided as a test-bed for
    > the technology. There is wiki page up outlining the new OSGI-based
    > plugin system and I am looking for feedback!:
    > http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle
    >
    >
    > OSGi is a great choice. Hazaa!
    >
    > Nick
    >
    > --
    > You received this message because you are subscribed to the Google Groups
    > "kettle-developers" group.
    > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > To unsubscribe from this group, send email to
    > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    > .
    > For more options, visit this group at
    > http://groups.google.com/group/kettle-developers?hl=en.
    >


    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  14. #14
    Jon Roberts Guest

    Default Re: Plugins and a market place

    Yes,

    Installations that are using Carte servers and a repository as an
    enterprise-level service could benefit from installation without a restart.

    On Tue, Nov 30, 2010 at 12:22 PM, Tony Cook <tony (AT) cloud2land (DOT) com> wrote:

    > Reading through the OSGI stuff and looking at the sample code, even though
    > I don't fully understand it, it looks well thought through and I can sense
    > the enthusiasm radiating from Orlando!
    >
    > I am just wondering if PDI really needs this ability to install modules
    > dynamically without a restart. Currently to install a plugin you just have
    > to copy the files (and any required jars) into the relevant plugin folder
    > and then restart Spoon. The benefit of a marketplace is to aid "discovery"
    > by users, but I'm sure a user would be more than happy to restart after
    > installing a plugin (like with Firefox).
    >
    > Unless there is a use-case of PDI that requires it to be permanently
    > running? Would this use case need to install/uninstall plugins at run time
    > with no possibility of a restart?
    >
    > Tony
    >
    >
    >
    > On Tue, Nov 30, 2010 at 1:52 AM, Nicholas Goodman <
    > ngoodman (AT) bayontechnologies (DOT) com> wrote:
    >
    >> On Nov 29, 2010, at 4:57 PM, codeoncoffee wrote:
    >>
    >> This effort is nowhere near complete and is provided as a test-bed for
    >> the technology. There is wiki page up outlining the new OSGI-based
    >> plugin system and I am looking for feedback!:
    >> http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle
    >>
    >>
    >> OSGi is a great choice. Hazaa!
    >>
    >> Nick
    >>
    >> --
    >> You received this message because you are subscribed to the Google Groups
    >> "kettle-developers" group.
    >> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    >> To unsubscribe from this group, send email to
    >> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    >> .
    >> For more options, visit this group at
    >> http://groups.google.com/group/kettle-developers?hl=en.
    >>

    >
    > --
    > You received this message because you are subscribed to the Google Groups
    > "kettle-developers" group.
    > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > To unsubscribe from this group, send email to
    > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    > .
    > For more options, visit this group at
    > http://groups.google.com/group/kettle-developers?hl=en.
    >




    --
    Sincerely,

    Jon Roberts

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  15. #15
    Peter Hunsberger Guest

    Default Re: Plugins and a market place

    On Tue, Nov 30, 2010 at 12:22 PM, Tony Cook <tony (AT) cloud2land (DOT) com> wrote:[color=blue]
    >
    > Reading through the OSGI stuff and looking at the sample code, even though I don't fully understand it, it looks well thought through and I can sense the enthusiasm radiating from Orlando!
    >
    > I am just wondering if PDI really needs this ability to install modules dynamically without a restart.

  16. #16
    codeoncoffee Guest

    Default Re: Plugins and a market place

    The ability to install without a restart is just a side-effect of the
    environment if properly setup. There is extra effort required to track
    these comings and goings and I agree that it's not necessary, despite
    the coolness factor

    -Nick Baker

    On Nov 30, 1:22

  17. #17
    dmoran Guest

    Default Re: Plugins and a market place

    Tony, I agree it might not be an absolute must have for spoon but it
    does add to the overall user experience. Especially if you are in the
    middle of developing a transform and realize you need a new step. It
    sure would be nice to be able to grab it and keep going. I'd like to
    see it as little more than another pallet to choose from. Another
    interesting case would be if you tried to open a transform that you
    didn't have the step or steps for, it would be sweet to have a prompt
    that says, "This transform requires the CoolAndNewStep, do you want to
    install it now? [yes] [no] [more info...]"

    We should strive for seamless, especially if it comes with the
    framework.

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  18. #18
    Matt Casters Guest

    Default Re: Plugins and a market place

    Kettle is traditionally open to change and open to contributions. This fact
    has not gone unnoticed even by our own developers at Pentaho.
    I like the fact that Kettle is being used to experiment with OSGi since it's
    a testament to a simple and straightforward architecture and I'm sure we'll
    all welcome the contribution the once it's done. Nick's aim is to make this
    not a replacement to the current plugin architecture (that Nick & I re-wrote
    in v4) but to be complimentary.

    However, Tony's point remains valid. OSGi is currently a technical solution
    looking for a problem that Kettle might not even have. If we would
    currently have a requirement to install a plugin without the need to restart
    Spoon, I'm sure it would fairly easy to do. The Kettle plugins are already
    dynamically loaded after all.

    In other words, let's retain your user experience requirements and add them
    to the rest and at the end we can see where we end up. Only then should we
    determine whether or not OSGi is needed for Kettle and/or for the rest of
    the Pentaho stack.

    In addition to the technical capabilities and the things we want as
    developers, there might be features that are not always desirable. One of
    the "features" I seriously dislike about certain frameworks (Eclipse, Talend
    and all) is the ability to automatically update. While technically a great
    idea, it presents some challenges as far as conformance and testing are
    concerned. Likewise, whatever plugin or market system we use, versioning is
    something we need to do very carefully. If we support automatic updates,
    allow for companies to turn it off or manage it centrally. Perhaps we want
    to support development, test, UAT and production-like environments for
    certain plugins? For other plugins a simple major, minor, point-version
    system would work fine. The plugins that are downloaded in Spoon, do we
    automatically want to deploy them in other locations too, signal Carte/DI
    Servers to download the same package? What about firewalls? Can we tell
    Spoon to download plugins from a shared folder or local server?

    There are many important questions that need to be asked and answered at
    this stage.

    If we don't ask these questions, and we would adopt Nicks work at this time,
    we would be putting a lot pressure on Nick to bend to the will of a
    constantly changing set of requirements and that hardly seems fair. All I'm
    saying is that the better the requirements we can gather are, before choices
    have been made, the easier it will be to phase the development and execute
    upon it later.

    Finally, it might make sense to have the exact same discussion in the rest
    of the Pentaho projects. I'm sure this would be a killer feature for the BI
    server & report designer as well.

    Thanks for caring!

    Matt


    2010/11/30 dmoran <dmoran (AT) pentaho (DOT) org>

    >
    > Tony, I agree it might not be an absolute must have for spoon but it
    > does add to the overall user experience. Especially if you are in the
    > middle of developing a transform and realize you need a new step. It
    > sure would be nice to be able to grab it and keep going. I'd like to
    > see it as little more than another pallet to choose from. Another
    > interesting case would be if you tried to open a transform that you
    > didn't have the step or steps for, it would be sweet to have a prompt
    > that says, "This transform requires the CoolAndNewStep, do you want to
    > install it now? [yes] [no] [more info...]"
    >
    > We should strive for seamless, especially if it comes with the
    > framework.
    >
    > --
    > You received this message because you are subscribed to the Google Groups
    > "kettle-developers" group.
    > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > To unsubscribe from this group, send email to
    > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    > .
    > For more options, visit this group at
    > http://groups.google.com/group/kettle-developers?hl=en.
    >
    >



    --
    Matt Casters <mcasters (AT) pentaho (DOT) org>
    Chief Data Integration, Kettle founder, Author of Pentaho Kettle
    Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
    (Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
    Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    Pentaho : The Commercial Open Source Alternative for Business Intelligence

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  19. #19
    dmoran Guest

    Default Re: Plugins and a market place

    I agree with Matt 108%

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  20. #20
    codeoncoffee Guest

    Default Re: Plugins and a market place

    I agree with you Matt. OSGI provides a wealth of options that aren't
    all desirable at the same time. Sitting down and deciding how to
    handle installation and configuration of modules across all of the
    platform is key to determining the direction to take.

    Last night I got the MaxMind step installed in Spoon by connecting to
    a Bundle Repository running on my box. It automatically installed the
    bundle I told it to and it's needed dependencies. Though this happened
    over the web it could have installed the files from a shared place on
    the filesystem or even a Maven repo. Pretty neat, plus I didn't write
    any code.

    How to handle system configuration for deployment is absolutely key.
    One of the great features of Apache Aries and Apache Karaf (just
    bundles you can add) is they allow you to work with "assemblies" of
    bundles. These can be locked configurations of the system installed in
    bulk. This may be a good approach to production deployment. There's
    just too many options to list and I'm discovering more as I continue
    this work.

    One thing to keep in mind with the OSGI environment is that our users
    can customize the entire deployment strategy by using existing OSGI
    bundles available in the community. The Karaf extensions allow you to
    manage SSHing directly into your running application and stop/start/
    install/etc. the running Bundles. I certainly wouldn't ship a system
    like this but could imagine wanting to do this as a user.

    I've been running Spoon with the Apache Felix Web Console which embeds
    a Jetty server running in your application with a web-app front-end to
    manage the system. Certainly a lot of overhead for configuring Spoon,
    but again, many options.

    I would love to schedule some time to discuss the needs of our
    individual products and the platform as a whole to try to find a
    unified approach. We were hoping to get further along and learn a lot
    more before presenting this initiative. I just couldn't wait anymore
    when I heard of this Marketplace idea!

    My immediate work will be on a walkthrough for creating a OSGI-
    provided steps focused on step developers, but if anyone has any
    specific questions or would like some pointers to examples, let me
    know.

    -Nick B

    On Nov 30, 2:25

  21. #21
    Nicholas Goodman Guest

    Default Re: Plugins and a market place

    On Nov 30, 2010, at 12:19 PM, codeoncoffee wrote:

    > I agree with you Matt. OSGI provides a wealth of options that aren't
    > all desirable at the same time. Sitting down and deciding how to


    With OSGi you get many things "for free" just by using it. The other thing one gets is lots of other smart people sorting out complicated issues like applications with depedencies across class loaders, etc.

    For instance, in a service engagement we're building a set of PDI Job plugins. The plugins all have some common functionality (base class) and then we extend them for the specific fcnality. Given the current plugin classloading mechanism, we can't actually install all this as a plugin.

    You can only have 1 plugin per plugin.xml, and each plugin has it's own classloader. Currently, you can't have a package (with a base class) either duplicated across all plugins (you'll get strange errors like Object X cannot be cast to Object X) or only in one (class not found from one plugin to another). The only way currently to solve this problem is to include the base classes, which are really part of the plugins, in the kettle libext and then have the individual plugins extended from that classloader.

    So... there's one piece of functionality that OSGi solves well. The ability for plugins to build "on top of each other" instead of just being able to build upon core Kettle. Enough to warrant a whole framework? Maybe not. All the same, use cases for the way OSGi handles these types of things are there.

    Nick

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  22. #22
    Matt Casters Guest

    Default Re: Plugins and a market place

    Hi Nick,

    Your assertions are fortunately not true.
    Have a look at the KFF source code (http://is.gd/iNJ0b) for an example on
    how you can easily deploy a series of plugins.
    In that particular case we deploy 8 plugins (job entries and steps) from a
    single jar file.
    Shared dependency jar files can be placed in an adjacent lib/ folder.
    And best of all: no XML is involved ;-)

    Cheers,

    Matt

    2010/12/15 Nicholas Goodman <ngoodman (AT) bayontechnologies (DOT) com>

    > On Nov 30, 2010, at 12:19 PM, codeoncoffee wrote:
    >
    > > I agree with you Matt. OSGI provides a wealth of options that aren't
    > > all desirable at the same time. Sitting down and deciding how to

    >
    > With OSGi you get many things "for free" just by using it. The other thing
    > one gets is lots of other smart people sorting out complicated issues like
    > applications with depedencies across class loaders, etc.
    >
    > For instance, in a service engagement we're building a set of PDI Job
    > plugins. The plugins all have some common functionality (base class) and
    > then we extend them for the specific fcnality. Given the current plugin
    > classloading mechanism, we can't actually install all this as a plugin.
    >
    > You can only have 1 plugin per plugin.xml, and each plugin has it's own
    > classloader. Currently, you can't have a package (with a base class) either
    > duplicated across all plugins (you'll get strange errors like Object X
    > cannot be cast to Object X) or only in one (class not found from one plugin
    > to another). The only way currently to solve this problem is to include the
    > base classes, which are really part of the plugins, in the kettle libext and
    > then have the individual plugins extended from that classloader.
    >
    > So... there's one piece of functionality that OSGi solves well. The
    > ability for plugins to build "on top of each other" instead of just being
    > able to build upon core Kettle. Enough to warrant a whole framework?
    > Maybe not. All the same, use cases for the way OSGi handles these types of
    > things are there.
    >
    > Nick
    >
    > --
    > You received this message because you are subscribed to the Google Groups
    > "kettle-developers" group.
    > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > To unsubscribe from this group, send email to
    > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    > .
    > For more options, visit this group at
    > http://groups.google.com/group/kettle-developers?hl=en.
    >
    >



    --
    Matt Casters <mcasters (AT) pentaho (DOT) org>
    Chief Data Integration, Kettle founder, Author of Pentaho Kettle
    Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
    (Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
    Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    Pentaho : The Commercial Open Source Alternative for Business Intelligence

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  23. #23
    Nicholas Goodman Guest

    Default Re: Plugins and a market place

    On Dec 15, 2010, at 11:05 AM, Matt Casters wrote:

    > Your assertions are fortunately not true.

    Love it when I'm wrong and it means a limitation isn't actually!!!

    > And best of all: no XML is involved ;-)

    Yeah... I saw some notes in the plugin loader that this was only for testing... Double checking looks like the general classpath search was the one that's marked as testing only. I got the sense that the annotations was the slightly off color method and that the plugin.xml was the primary way to build plugins. Are annotations now the preferred method?

    Does the annotation method bring in all annotated plugins into separate classloaders? ie, with XML you get one classloader per plugin. Now, it would seem you get either
    1) a single classloader for all annotated plugins
    -or-
    2) a single classloader per loaded .jar in plugins/ dir

    This is great news! Makes my day! Multiple plugins in a single jar!

    Nick

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  24. #24
    Matt Casters Guest

    Default Re: Plugins and a market place

    Nick, there is no "preferred" method as we decided to support both methods
    of plugin development.
    However, since v4 it's obviously a lot more convenient to use annotations
    for its ease of use.
    There was a slight drawback to developing in Eclipse as documented over
    here: http://is.gd/iNMn1
    However, even that beats restarting Spoon a few hundred times to get a
    plugin developed ;-)
    The ease of use of just being able to unzip a file in plugins/ or drop a jar
    file in there was worth the effort we put into it I think.

    Plugins are added to the collective by whichever way works for you:
    plugin.xml, annotations, kettle-steps.xml, kettle-jobentries.xml and one day
    OSGi. From there on they play their role the same like all steps and job
    entries.
    And yes, they do indeed all use separate class loaders (Except for the SAP
    Input step I believe)

    Cheers,
    Matt

    2010/12/15 Nicholas Goodman <ngoodman (AT) bayontechnologies (DOT) com>

    >
    > On Dec 15, 2010, at 11:05 AM, Matt Casters wrote:
    >
    > > Your assertions are fortunately not true.

    > Love it when I'm wrong and it means a limitation isn't actually!!!
    >
    > > And best of all: no XML is involved ;-)

    > Yeah... I saw some notes in the plugin loader that this was only for
    > testing... Double checking looks like the general classpath search was the
    > one that's marked as testing only. I got the sense that the annotations was
    > the slightly off color method and that the plugin.xml was the primary way to
    > build plugins. Are annotations now the preferred method?
    >
    > Does the annotation method bring in all annotated plugins into separate
    > classloaders? ie, with XML you get one classloader per plugin. Now, it
    > would seem you get either
    > 1) a single classloader for all annotated plugins
    > -or-
    > 2) a single classloader per loaded .jar in plugins/ dir
    >
    > This is great news! Makes my day! Multiple plugins in a single jar!
    >
    > Nick
    >
    > --
    > You received this message because you are subscribed to the Google Groups
    > "kettle-developers" group.
    > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    > To unsubscribe from this group, send email to
    > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
    > .
    > For more options, visit this group at
    > http://groups.google.com/group/kettle-developers?hl=en.
    >
    >



    --
    Matt Casters <mcasters (AT) pentaho (DOT) org>
    Chief Data Integration, Kettle founder, Author of Pentaho Kettle
    Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
    (Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
    Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
    Pentaho : The Commercial Open Source Alternative for Business Intelligence

    --
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

  25. #25
    codeoncoffee Guest

    Default Re: Plugins and a market place

    Quick update. I've just committed a revised approach to getting OSGI
    in Kettle. This new implementation is completely hidden behind the
    current PluginRegistry api. With this commit, OSGI is just another
    avenue to getting your plugin registered with the current system.

    I will be adding a listener mechanism to the PluginRegistry so Kettle
    can become aware as plugins come and go and react appropriately. This
    provides the hot-swapping and loading.

    The rest of the functionality added by OSGI will be available to the
    plugin developers through traditional OSGI APIs. The MaxMind example
    step uses the Blueprint beans approach right now. So far I'm only
    registering Spoon as a service object but others will surely come.

    I'm reasonably satisfied with this current effort. I'd like to see
    more types proven-out other than steps, scheduler interface for
    instance would be a nice demo. And the current directory watching
    system needs to be replaced with Apache Karaf or just removed.

    -Nick B.

    On Dec 15, 2:28

  26. #26
    codeoncoffee Guest

    Default Re: Plugins and a market place

    The wiki has been updated with samples from the MaxMind Plugin:
    http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle



    On Dec 21, 8:55

  27. #27
    codeoncoffee Guest

    Default Re: Plugins and a market place

    The wiki has been updated with samples from the MaxMind Plugin:
    http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle

    On Dec 21, 8:55

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.