Plugins and a market place

[Matt Casters]

Dear Kettle devs,

For a while now we've been thinking about creating a market place for
Pentaho related plugins.
The constant influx of new steps, job entries and Spoon plugins makes it
increasingly hard to do proper dependency management (see previous mail).
One solution could be to push away more and more of these steps and job
entries to external plugins. However, we fear that developers might see
those as second class citizens compared to "core" components of
Kettle. Developers might think that for the following reasons:
1) Plugins are listed on a wiki page but otherwise not structurally
described.
2) Installation of the plugins is manual process and technical of nature,
you have to take into account versions, and so on.
3) Communication methods with the plugin authors (e-mail, JIRA, forum) are
not always clear. (usually not!)
4) The plugins are not maintained (code fixes, translation/i18n, code
migration, ...) the same way as the Kettle core codebase.

I strongly think that having a market in place that would do away with all
these problems would in fact promote the creation of new plugins even more.

In short, Pentaho is looking for ways to make it easier to integrate new
functionality in Kettle without having to resort to the monolithic structure
we have in place now. Any ideas you might have in this respect are most
welcome.

Regards,

Matt
--
Matt Casters <mcasters (AT) pentaho (DOT) org>
Chief Data Integration, Kettle founder, Author of Pentaho Kettle
Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
(Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
Pentaho : The Commercial Open Source Alternative for Business Intelligence

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Roland Bouman]

Hi All!

Overall, a good idea.
I do think that to make it work, there'd have to be clear admission
criteria, and an independent board / comittee to judge submissions
against these criteria.
Feedback to devs should also be in terms of those criteria in case
their submission is currently not admitted.

One of the criteria I would suggest is availability of freely
available documentation (so it can be linked to from the marketplace
site).
I would also suggest criteria for the content and structure of the
documentation.
The value in this is to make it easy for potential users to discover
whether it makes sense for them to use a a particular plugin.

kind regards,

Roland

On Mon, Nov 29, 2010 at 1:45 PM, Matt Casters <mcasters (AT) pentaho (DOT) org> wrote:[color=blue]
> Dear Kettle devs,
> For a while now we've been thinking about creating a market place for
> Pentaho related plugins.
> The constant influx of new steps, job entries and Spoon plugins makes it
> increasingly hard to do proper dependency management (see previous mail).
> One solution could be to push away more and more of these steps and job
> entries to external plugins.

[Matt Casters]

Hi Roland,

I'm not sure I would want to be restrictive at all. I want people to have
as much fun with the plugins as possible. I *want* to have experimental,
undocumented stuff in the market. I want that since I strongly believe
that's how open source comes to the best results. However, I also would
expect to have a system in place that would allow me to vote plugins up or
down from within Spoon, integrated problem reporting &JIRA, mini-forums for
feedback, and so on.

Having integrated documentation however is a good point. Giving a plugin a
dedicated wiki page to drop the docs into is a good idea. Forcing people to
document is IMO over the top. Perhaps we should create a "Vote down: no
documentation" option somewhere.

All in all, if someone contributes a plugin to calculate Pi in arbitrary
position to 1M decimals, we can safely say that this is something we would
like to keep as a plugin because the use-case is a bit scientific/specific.
However, for a lot of other plugins that are imaginable in the data
generation, data quality, data preparation areas, that's not clear at all.
Myself nor the folks at Pentaho, nor any committee or board I can think of
would want to make that decision either.

Personally I kinda like the Android market idea where folks can rate and
review apps. Perhaps we can combine that open "anyone can contribute" idea
with a lot of infrastructure to facilitate two-way-communications,
development, translation, documentation and source code access.

Just my two Euro-cents,
Matt

2010/11/29 Roland Bouman <roland.bouman (AT) gmail (DOT) com>

> Hi All!
>
> Overall, a good idea.
> I do think that to make it work, there'd have to be clear admission
> criteria, and an independent board / comittee to judge submissions
> against these criteria.
> Feedback to devs should also be in terms of those criteria in case
> their submission is currently not admitted.
>
> One of the criteria I would suggest is availability of freely
> available documentation (so it can be linked to from the marketplace
> site).
> I would also suggest criteria for the content and structure of the
> documentation.
> The value in this is to make it easy for potential users to discover
> whether it makes sense for them to use a a particular plugin.
>
> kind regards,
>
> Roland
>
> On Mon, Nov 29, 2010 at 1:45 PM, Matt Casters <mcasters (AT) pentaho (DOT) org>
> wrote:
> > Dear Kettle devs,
> > For a while now we've been thinking about creating a market place for
> > Pentaho related plugins.
> > The constant influx of new steps, job entries and Spoon plugins makes it
> > increasingly hard to do proper dependency management (see previous mail).
> > One solution could be to push away more and more of these steps and job
> > entries to external plugins. However, we fear that developers might see
> > those as second class citizens compared to "core" components of
> > Kettle. Developers might think that for the following reasons:
> > 1) Plugins are listed on a wiki page but otherwise not structurally
> > described.
> > 2) Installation of the plugins is manual process and technical of nature,
> > you have to take into account versions, and so on.
> > 3) Communication methods with the plugin authors (e-mail, JIRA, forum)
> are
> > not always clear. (usually not!)
> > 4) The plugins are not maintained (code fixes, translation/i18n, code
> > migration, ...) the same way as the Kettle core codebase.
> > I strongly think that having a market in place that would do away with
> all
> > these problems would in fact promote the creation of new plugins even
> more.
> > In short, Pentaho is looking for ways to make it easier to integrate new
> > functionality in Kettle without having to resort to
> the monolithic structure
> > we have in place now. Any ideas you might have in this respect are most
> > welcome.
> > Regards,
> >
> > Matt
> > --
> > Matt Casters <mcasters (AT) pentaho (DOT) org>
> > Chief Data Integration, Kettle founder, Author of Pentaho Kettle
> > Solutions (Wiley)
> > Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
> > Pentaho : The Commercial Open Source Alternative for Business
> Intelligence
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "kettle-developers" group.
> > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> > To unsubscribe from this group, send email to
> > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> > For more options, visit this group at
> > http://groups.google.com/group/kettle-developers?hl=en.
> >
>
>
>
> --
> Roland Bouman
> blog: http://rpbouman.blogspot.com/
> twitter: @rolandbouman
>
> Author of "Pentaho Solutions: Business Intelligence and Data
> Warehousing with Pentaho and MySQL",
> http://tinyurl.com/lvxa88 (Wiley, ISBN: 978-0-470-48432-6)
>
> Author of "Pentaho Kettle Solutions: Building Open Source ETL
> Solutions with Pentaho Data Integration",
> http://tinyurl.com/33r7a8m (Wiley, ISBN: 978-0-470-63517-9)
>
> --
> You received this message because you are subscribed to the Google Groups
> "kettle-developers" group.
> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> To unsubscribe from this group, send email to
> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> For more options, visit this group at
> http://groups.google.com/group/kettle-developers?hl=en.
>
>


--
Matt Casters <mcasters (AT) pentaho (DOT) org>
Chief Data Integration, Kettle founder, Author of Pentaho Kettle
Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
(Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
Pentaho : The Commercial Open Source Alternative for Business Intelligence

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Roland Bouman]

Hi Matt, all,

you raise a number of good points I hadn't thought about.

My reply mostly reflects my own experiences hunting for plugins.

With a few exceptions (web browser plugins), whether its CMS-related
(drupal), web development (jQuery), I find I spend oodles of time
downloading and trying plugins just to find out what they actually do.
In many cases, documentation is so sparse that it's hard to even
comprehend what the function of the plugin is. Just a little bit of
documentation (screenshot + typical example) would remedey say 50% of
those problems.

But indeed maybe documentation criteria is a bit over the top. The
voting system may indeed work as a filter - thanks for pointing that
out.

regards,

Roland

On Mon, Nov 29, 2010 at 2:05 PM, Matt Casters <mcasters (AT) pentaho (DOT) org> wrote:[color=blue]
> Hi Roland,
> I'm not sure I would want to be restrictive at all.

[Brandon Jackson]

You'll have to catch the plugin at the source where developers feed for
information. That could be a dev list registration like this, or the
pentaho forums. If there was a simple mechanism to 'request' a GUID for
use in a plugin, then you could get an idea of how many are being attempted.
The same GUID combined with a plugin.xml could give some catalog of
plugins. The XML file could have a URLs inside that show a preferred link
where 'loved' plugins could reside with a pentaho controlled URL, other URLs
like 'author home' could point to at least another source.

If each plugin had a GUID, then maybe an option in spoon would be to scan
and report which GUIDs are used. A context menu on a step or job and allow
voting up or down on versions of plugins, much like iTunes.

It would probably be cool if there were a utility that could scan all the
jobs and transformations just collecting the GUIDs and occurrence counts.
(Popularity). You would especially want to make this option available on
first run of a new version. (May reveal cracks before first run of a new
version.)


On Mon, Nov 29, 2010 at 7:23 AM, Roland Bouman <roland.bouman (AT) gmail (DOT) com>wrote:

> Hi Matt, all,
>
> you raise a number of good points I hadn't thought about.
>
> My reply mostly reflects my own experiences hunting for plugins.
>
> With a few exceptions (web browser plugins), whether its CMS-related
> (drupal), web development (jQuery), I find I spend oodles of time
> downloading and trying plugins just to find out what they actually do.
> In many cases, documentation is so sparse that it's hard to even
> comprehend what the function of the plugin is. Just a little bit of
> documentation (screenshot + typical example) would remedey say 50% of
> those problems.
>
> But indeed maybe documentation criteria is a bit over the top. The
> voting system may indeed work as a filter - thanks for pointing that
> out.
>
> regards,
>
> Roland
>
> On Mon, Nov 29, 2010 at 2:05 PM, Matt Casters <mcasters (AT) pentaho (DOT) org>
> wrote:
> > Hi Roland,
> > I'm not sure I would want to be restrictive at all. I want people to
> have
> > as much fun with the plugins as possible. I want to have experimental,
> > undocumented stuff in the market. I want that since I strongly believe
> > that's how open source comes to the best results. However, I also would
> > expect to have a system in place that would allow me to vote plugins up
> or
> > down from within Spoon, integrated problem reporting &JIRA, mini-forums
> for
> > feedback, and so on.
> > Having integrated documentation however is a good point. Giving a plugin
> a
> > dedicated wiki page to drop the docs into is a good idea. Forcing people
> to
> > document is IMO over the top. Perhaps we should create a "Vote down: no
> > documentation" option somewhere.
> > All in all, if someone contributes a plugin to calculate Pi in arbitrary
> > position to 1M decimals, we can safely say that this is something we
> would
> > like to keep as a plugin because the use-case is a bit
> scientific/specific.
> > However, for a lot of other plugins that are imaginable in the data
> > generation, data quality, data preparation areas, that's not clear at
> all.
> > Myself nor the folks at Pentaho, nor any committee or board I can think
> of
> > would want to make that decision either.
> > Personally I kinda like the Android market idea where folks can rate and
> > review apps. Perhaps we can combine that open "anyone can contribute"
> idea
> > with a lot of infrastructure to facilitate two-way-communications,
> > development, translation, documentation and source code access.
> > Just my two Euro-cents,
> > Matt
> >
> > 2010/11/29 Roland Bouman <roland.bouman (AT) gmail (DOT) com>
> >>
> >> Hi All!
> >>
> >> Overall, a good idea.
> >> I do think that to make it work, there'd have to be clear admission
> >> criteria, and an independent board / comittee to judge submissions
> >> against these criteria.
> >> Feedback to devs should also be in terms of those criteria in case
> >> their submission is currently not admitted.
> >>
> >> One of the criteria I would suggest is availability of freely
> >> available documentation (so it can be linked to from the marketplace
> >> site).
> >> I would also suggest criteria for the content and structure of the
> >> documentation.
> >> The value in this is to make it easy for potential users to discover
> >> whether it makes sense for them to use a a particular plugin.
> >>
> >> kind regards,
> >>
> >> Roland
> >>
> >> On Mon, Nov 29, 2010 at 1:45 PM, Matt Casters <mcasters (AT) pentaho (DOT) org>
> >> wrote:
> >> > Dear Kettle devs,
> >> > For a while now we've been thinking about creating a market place for
> >> > Pentaho related plugins.
> >> > The constant influx of new steps, job entries and Spoon plugins makes
> it
> >> > increasingly hard to do proper dependency management (see previous
> >> > mail).
> >> > One solution could be to push away more and more of these steps and
> job
> >> > entries to external plugins. However, we fear that developers might
> see
> >> > those as second class citizens compared to "core" components of
> >> > Kettle. Developers might think that for the following reasons:
> >> > 1) Plugins are listed on a wiki page but otherwise not structurally
> >> > described.
> >> > 2) Installation of the plugins is manual process and technical of
> >> > nature,
> >> > you have to take into account versions, and so on.
> >> > 3) Communication methods with the plugin authors (e-mail, JIRA, forum)
> >> > are
> >> > not always clear. (usually not!)
> >> > 4) The plugins are not maintained (code fixes, translation/i18n, code
> >> > migration, ...) the same way as the Kettle core codebase.
> >> > I strongly think that having a market in place that would do away with
> >> > all
> >> > these problems would in fact promote the creation of new plugins even
> >> > more.
> >> > In short, Pentaho is looking for ways to make it easier to integrate
> new
> >> > functionality in Kettle without having to resort to
> >> > the monolithic structure
> >> > we have in place now. Any ideas you might have in this respect are
> most
> >> > welcome.
> >> > Regards,
> >> >
> >> > Matt
> >> > --
> >> > Matt Casters <mcasters (AT) pentaho (DOT) org>
> >> > Chief Data Integration, Kettle founder, Author of Pentaho Kettle
> >> > Solutions (Wiley)
> >> > Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
> >> > Pentaho : The Commercial Open Source Alternative for Business
> >> > Intelligence
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "kettle-developers" group.
> >> > To post to this group, send email to
> kettle-developers (AT) googlegroups (DOT) com.
> >> > To unsubscribe from this group, send email to
> >> > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/kettle-developers?hl=en.
> >> >
> >>
> >>
> >>
> >> --
> >> Roland Bouman
> >> blog: http://rpbouman.blogspot.com/
> >> twitter: @rolandbouman
> >>
> >> Author of "Pentaho Solutions: Business Intelligence and Data
> >> Warehousing with Pentaho and MySQL",
> >> http://tinyurl.com/lvxa88 (Wiley, ISBN: 978-0-470-48432-6)
> >>
> >> Author of "Pentaho Kettle Solutions: Building Open Source ETL
> >> Solutions with Pentaho Data Integration",
> >> http://tinyurl.com/33r7a8m (Wiley, ISBN: 978-0-470-63517-9)
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "kettle-developers" group.
> >> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
> .
> >> To unsubscribe from this group, send email to
> >> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> >> For more options, visit this group at
> >> http://groups.google.com/group/kettle-developers?hl=en.
> >>
> >
> >
> >
> > --
> > Matt Casters <mcasters (AT) pentaho (DOT) org>
> > Chief Data Integration, Kettle founder, Author of Pentaho Kettle
> > Solutions (Wiley)
> > Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
> > Pentaho : The Commercial Open Source Alternative for Business
> Intelligence
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "kettle-developers" group.
> > To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> > To unsubscribe from this group, send email to
> > kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> > For more options, visit this group at
> > http://groups.google.com/group/kettle-developers?hl=en.
> >
>
>
>
> --
> Roland Bouman
> blog: http://rpbouman.blogspot.com/
> twitter: @rolandbouman
>
> Author of "Pentaho Solutions: Business Intelligence and Data
> Warehousing with Pentaho and MySQL",
> http://tinyurl.com/lvxa88 (Wiley, ISBN: 978-0-470-48432-6)
>
> Author of "Pentaho Kettle Solutions: Building Open Source ETL
> Solutions with Pentaho Data Integration",
> http://tinyurl.com/33r7a8m (Wiley, ISBN: 978-0-470-63517-9)
>
> --
> You received this message because you are subscribed to the Google Groups
> "kettle-developers" group.
> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> To unsubscribe from this group, send email to
> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> For more options, visit this group at
> http://groups.google.com/group/kettle-developers?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Jon Roberts]

While a plugin marketplace could help with some of the dependency issues
with better documentation, I suspect that the issue will persist.

A quick search reveals what you might expect. Dependency problems are not
unique to kettle.
http://www.google.com/search?rlz=1C1...age+dependency


<http://www.google.com/search?rlz=1C1GPCK_enUS342US342&sourceid=chrome&ie=UTF-8&q=eclipse+package+dependency>Some
type of packaging, central repositories, and associated management tools are
usually employed to handle this. There are lots of examples for both OS
(rpm, dpk) and programming environments (CPAN, pip - eggs - pypm). Given
the widespread adoption of java and eclipse both, I'm somewhat surprised
that a standard has not already emerged for this? Maybe there is one and
I'm just unaware of it?

On Mon, Nov 29, 2010 at 6:45 AM, Matt Casters <mcasters (AT) pentaho (DOT) org> wrote:

> Dear Kettle devs,
>
> For a while now we've been thinking about creating a market place for
> Pentaho related plugins.
> The constant influx of new steps, job entries and Spoon plugins makes it
> increasingly hard to do proper dependency management (see previous mail).
> One solution could be to push away more and more of these steps and job
> entries to external plugins. However, we fear that developers might see
> those as second class citizens compared to "core" components of
> Kettle. Developers might think that for the following reasons:
> 1) Plugins are listed on a wiki page but otherwise not structurally
> described.
> 2) Installation of the plugins is manual process and technical of nature,
> you have to take into account versions, and so on.
> 3) Communication methods with the plugin authors (e-mail, JIRA, forum) are
> not always clear. (usually not!)
> 4) The plugins are not maintained (code fixes, translation/i18n, code
> migration, ...) the same way as the Kettle core codebase.
>
> I strongly think that having a market in place that would do away with all
> these problems would in fact promote the creation of new plugins even more.
>
> In short, Pentaho is looking for ways to make it easier to integrate new
> functionality in Kettle without having to resort to the monolithic structure
> we have in place now. Any ideas you might have in this respect are most
> welcome.
>
> Regards,
>
> Matt
> --
> Matt Casters <mcasters (AT) pentaho (DOT) org>
> Chief Data Integration, Kettle founder, Author of Pentaho Kettle Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
> (Wiley<http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>
> )
> Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
> Pentaho : The Commercial Open Source Alternative for Business Intelligence
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "kettle-developers" group.
> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> To unsubscribe from this group, send email to
> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> For more options, visit this group at
> http://groups.google.com/group/kettle-developers?hl=en.
>



--
Sincerely,

Jon Roberts

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Matt Casters]

Actually, Kettle plugins are loaded by separate class loaders. As such, a
solid plugin system like a market would indeed lower dependencies since you
don't have to ship with libfoo-2.1.3.jar in the main Kettle distribution.
The jar file gets isolated and is loaded for a single specific plugin.

We don't have to overstate the dependency issues we have now either.
Usually these are minor things like upgrading a commons library from 3.1.2
to 3.1.3 or going from POI 3.6 to 3.7. The problem is mainly located in the
area of *knowing* which libraries are in conflict with other projects, not
the resolution as such. For example, you want to have an answer to
questions like: I want to get data from a Kettle transformation in report
designer. Which libraries do I need, which of those do I already have in
PRD and which are in conflict?

Take care,

Matt

2010/11/29 Jon Roberts <bigjonroberts (AT) gmail (DOT) com>

> While a plugin marketplace could help with some of the dependency issues
> with better documentation, I suspect that the issue will persist.
>
> A quick search reveals what you might expect. Dependency problems are not
> unique to kettle.
>
> http://www.google.com/search?rlz=1C1...age+dependency
>
>
>
> <http://www.google.com/search?rlz=1C1GPCK_enUS342US342&sourceid=chrome&ie=UTF-8&q=eclipse+package+dependency>Some
> type of packaging, central repositories, and associated management tools are
> usually employed to handle this. There are lots of examples for both OS
> (rpm, dpk) and programming environments (CPAN, pip - eggs - pypm). Given
> the widespread adoption of java and eclipse both, I'm somewhat surprised
> that a standard has not already emerged for this? Maybe there is one and
> I'm just unaware of it?
>
> On Mon, Nov 29, 2010 at 6:45 AM, Matt Casters <mcasters (AT) pentaho (DOT) org>wrote:
>
>> Dear Kettle devs,
>>
>> For a while now we've been thinking about creating a market place for
>> Pentaho related plugins.
>> The constant influx of new steps, job entries and Spoon plugins makes it
>> increasingly hard to do proper dependency management (see previous mail).
>> One solution could be to push away more and more of these steps and job
>> entries to external plugins. However, we fear that developers might see
>> those as second class citizens compared to "core" components of
>> Kettle. Developers might think that for the following reasons:
>> 1) Plugins are listed on a wiki page but otherwise not structurally
>> described.
>> 2) Installation of the plugins is manual process and technical of nature,
>> you have to take into account versions, and so on.
>> 3) Communication methods with the plugin authors (e-mail, JIRA, forum) are
>> not always clear. (usually not!)
>> 4) The plugins are not maintained (code fixes, translation/i18n, code
>> migration, ...) the same way as the Kettle core codebase.
>>
>> I strongly think that having a market in place that would do away with all
>> these problems would in fact promote the creation of new plugins even more.
>>
>> In short, Pentaho is looking for ways to make it easier to integrate new
>> functionality in Kettle without having to resort to the monolithic structure
>> we have in place now. Any ideas you might have in this respect are most
>> welcome.
>>
>> Regards,
>>
>> Matt
>> --
>> Matt Casters <mcasters (AT) pentaho (DOT) org>
>> Chief Data Integration, Kettle founder, Author of Pentaho Kettle
>> Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
>> (Wiley<http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>
>> )
>> Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
>> Pentaho : The Commercial Open Source Alternative for Business Intelligence
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "kettle-developers" group.
>> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
>> To unsubscribe from this group, send email to
>> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/kettle-developers?hl=en.
>>
>
>
>
> --
> Sincerely,
>
> Jon Roberts
>
> --
> You received this message because you are subscribed to the Google Groups
> "kettle-developers" group.
> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> To unsubscribe from this group, send email to
> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> For more options, visit this group at
> http://groups.google.com/group/kettle-developers?hl=en.
>



--
Matt Casters <mcasters (AT) pentaho (DOT) org>
Chief Data Integration, Kettle founder, Author of Pentaho Kettle
Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
(Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
Pentaho : The Commercial Open Source Alternative for Business Intelligence

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Tony Cook]

Now this is a good idea! The opportunities for PDI as a

[Mark Hall]

The Weka project went through just this exercise with the last release
of the development version in late July. I introduced a new package
management system, based heavily on what the R statistical software
uses. All contributions were split out of the core of Weka along with
many core algorithms and made into packages. The whole shebang is
hosted on Sourceforge - this includes a central package meta data
repository and the actual packages themselves. Weka allows for
dependencies between not only the base system but other packages too
(something that Kettle doesn't concern itself with given its separate
class loader per plugin approach) and the package management system
takes care of handling dependencies and detecting conflicts. Users can
use either a command line or GUI client to download and install
packages.

We have a concept of "official" and "unofficial" packages for Weka.
Official packages get their meta data hosted at the central repository
(the package itself can reside anywhere that is http accessible) and
are verified by the Weka team (to the extent that the package
functions and does not contain malware etc.). Unofficial packages
don't get their meta data hosted at the central repository and can be
installed by just pointing the package manager at their URL. The
unofficial route allows folks to quickly make (possibly) experimental
stuff available without having to go through the official channel.

The main motivation for the Weka project was to reduce the burden of
supporting external contributions. It makes a clear delineation
between what is provided by the core Weka team and what is a
contribution. A secondary goal was to open up Weka to the use of
third-party libraries (something that has been avoided like the plague
in the past :-)). Third-party library related issues/problems are now
clearly the domain of package contributers who include them in their
packages.

This has all been working just swimmingly since the release. Packages
have been downloaded via the package manager more than 33,000 times
since late July.

The package manager code is independent of Weka and is checked in to
Pentaho's public SVN.

For those who are interested, the central repository is at:
http://weka.sourceforge.net/packageMetaData/

Information on the clients and package structure is at:
http://weka.wikispaces.com/How+do+I+...age+manager%3F
http://weka.wikispaces.com/How+are+p...ment+system%3F

Cheers,
Mark.


On Tue, Nov 30, 2010 at 1:51 AM, Roland Bouman <roland.bouman (AT) gmail (DOT) com> wrote:[color=blue]
> Hi All!
>
> Overall, a good idea.
> I do think that to make it work, there'd have to be clear admission
> criteria, and an independent board / comittee to judge submissions
> against these criteria.
> Feedback to devs should also be in terms of those criteria in case
> their submission is currently not admitted.
>
> One of the criteria I would suggest is availability of freely
> available documentation (so it can be linked to from the marketplace
> site).
> I would also suggest criteria for the content and structure of the
> documentation.
> The value in this is to make it easy for potential users to discover
> whether it makes sense for them to use a a particular plugin.
>
> kind regards,
>
> Roland
>
> On Mon, Nov 29, 2010 at 1:45 PM, Matt Casters <mcasters (AT) pentaho (DOT) org> wrote:[color=green]
>> Dear Kettle devs,
>> For a while now we've been thinking about creating a market place for
>> Pentaho related plugins.
>> The constant influx of new steps, job entries and Spoon plugins makes it
>> increasingly hard to do proper dependency management (see previous mail)..
>> One solution could be to push away more and more of these steps and job
>> entries to external plugins.

[codeoncoffee]

Actually there is a standard for this. It's the underpinning of the
Eclipse plugin system, OSGI:
http://en.wikipedia.org/wiki/OSGi
http://www.osgi.org/About/WhyOSGi

The Eclipse project migrated to OSGI in v3 and maintains what's
considered the reference implementation in Equinox ( http://www.eclipse.org/equinox/
). Recently many other companies and projects have migrated to OSGI
themselves or provided support for it including: JBoss, WebShere,
GlassFish, Netbeans, Weblogic, SpringSource, Apache Sling... and the
list goes on.

OSGI provides a component architecture framework for applications
based on plain Jars with runtime dependency information contained as
metadata in the jar manifests. The OSGI runtime manages wiring up the
component's (Bundles) with needed library classes and provides several
ways for services to be shared and tracked between bundles.

There's even built-in support for downloading dependencies over the
internet via a Bundle Repository (OBR) such as SpringSource's (
http://ebr.springsource.com/repository/app/bundle ) or though a Maven
repository. If you've installed a plugin in Eclipse you've seen OSGI
in action. You can install/start/stop/restart a plugin without
bringing down the application.

There's quite a large OSGI community out there with a great number of
bundles to take advantage of and resources for developers. And they're
not light-weights. Last I checked there are around 10 JSRs based on
OSGI. All of the major IDEs support development of Bundles and there
excellent plugins for Ant and Maven to support building. But as I
mentioned, they're just Jars. It doesn't take much to get started.

If you can't tell I'm a big proponent . Pentaho has large number of
plugin/service/configuration systems, all different and incomplete. As
we further integrate our products we're duplicating a lot of
integration code (plugin wiring, initialization, configuration) and
maintaining and extending the various systems in parallel. All that
could be standardized under a unified component (plugin) system.

For the past couple months I've been leading a group here
investigating just this thing. We decided Kettle would be the natural
place to investigate an OSGI migration. It's plugin system is quite
similar in many ways and it has a very rich component set with many
plugins. Also it touches just about all of our technologies and
projects I just recently checked-in the first code:

OSGI-ified version of Kettle with preliminary support for Bundle
provided steps checked-in at:
(svn://source.pentaho.org/svnkettleroot/Kettle/branches/osgi)

This past week I worked on converting over the MaxMind Lookup step.
It's also checked-in:
(svn://source.pentaho.org/svnkettleroot/plugins/MaxMindGeoIPLookup/
branches/osgi)

This effort is nowhere near complete and is provided as a test-bed for
the technology. There is wiki page up outlining the new OSGI-based
plugin system and I am looking for feedback!:
http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle

While this is all preliminary, I'm scheduled to host a brown-bag for
the Orlando developers sharing what I've learning. Doug Moran has also
asked me to present this formally to the Kettle Devs in an upcoming
community WebEx as a potential way forward.

Cheers,
Nick Baker



On Nov 29, 9:48

[Nicholas Goodman]

On Nov 29, 2010, at 4:57 PM, codeoncoffee wrote:

> This effort is nowhere near complete and is provided as a test-bed for
> the technology. There is wiki page up outlining the new OSGI-based
> plugin system and I am looking for feedback!:
> http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle

OSGi is a great choice. Hazaa!

Nick

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[dmoran]

For the popularity issue I think the more we can automate the better.
We could add a heartbeat that reports stats back and is available with
the download that shows how many times it has been installed,
uninstalled and run - that would also help with the niche type plugins
where total downloads is small, but people who do use it - use it a
lot. The key to success in something like this is automation and low
touch.

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Tony Cook]

Reading through the OSGI stuff and looking at the sample code, even though I
don't fully understand it, it looks well thought through and I can sense the
enthusiasm radiating from Orlando!

I am just wondering if PDI really needs this ability to install modules
dynamically without a restart. Currently to install a plugin you just have
to copy the files (and any required jars) into the relevant plugin folder
and then restart Spoon. The benefit of a marketplace is to aid "discovery"
by users, but I'm sure a user would be more than happy to restart after
installing a plugin (like with Firefox).

Unless there is a use-case of PDI that requires it to be permanently
running? Would this use case need to install/uninstall plugins at run time
with no possibility of a restart?

Tony


On Tue, Nov 30, 2010 at 1:52 AM, Nicholas Goodman <
ngoodman (AT) bayontechnologies (DOT) com> wrote:

> On Nov 29, 2010, at 4:57 PM, codeoncoffee wrote:
>
> This effort is nowhere near complete and is provided as a test-bed for
> the technology. There is wiki page up outlining the new OSGI-based
> plugin system and I am looking for feedback!:
> http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle
>
>
> OSGi is a great choice. Hazaa!
>
> Nick
>
> --
> You received this message because you are subscribed to the Google Groups
> "kettle-developers" group.
> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> To unsubscribe from this group, send email to
> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> For more options, visit this group at
> http://groups.google.com/group/kettle-developers?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Jon Roberts]

Yes,

Installations that are using Carte servers and a repository as an
enterprise-level service could benefit from installation without a restart.

On Tue, Nov 30, 2010 at 12:22 PM, Tony Cook <tony (AT) cloud2land (DOT) com> wrote:

> Reading through the OSGI stuff and looking at the sample code, even though
> I don't fully understand it, it looks well thought through and I can sense
> the enthusiasm radiating from Orlando!
>
> I am just wondering if PDI really needs this ability to install modules
> dynamically without a restart. Currently to install a plugin you just have
> to copy the files (and any required jars) into the relevant plugin folder
> and then restart Spoon. The benefit of a marketplace is to aid "discovery"
> by users, but I'm sure a user would be more than happy to restart after
> installing a plugin (like with Firefox).
>
> Unless there is a use-case of PDI that requires it to be permanently
> running? Would this use case need to install/uninstall plugins at run time
> with no possibility of a restart?
>
> Tony
>
>
>
> On Tue, Nov 30, 2010 at 1:52 AM, Nicholas Goodman <
> ngoodman (AT) bayontechnologies (DOT) com> wrote:
>
>> On Nov 29, 2010, at 4:57 PM, codeoncoffee wrote:
>>
>> This effort is nowhere near complete and is provided as a test-bed for
>> the technology. There is wiki page up outlining the new OSGI-based
>> plugin system and I am looking for feedback!:
>> http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle
>>
>>
>> OSGi is a great choice. Hazaa!
>>
>> Nick
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "kettle-developers" group.
>> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
>> To unsubscribe from this group, send email to
>> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/kettle-developers?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "kettle-developers" group.
> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> To unsubscribe from this group, send email to
> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> For more options, visit this group at
> http://groups.google.com/group/kettle-developers?hl=en.
>



--
Sincerely,

Jon Roberts

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Peter Hunsberger]

On Tue, Nov 30, 2010 at 12:22 PM, Tony Cook <tony (AT) cloud2land (DOT) com> wrote:[color=blue]
>
> Reading through the OSGI stuff and looking at the sample code, even though I don't fully understand it, it looks well thought through and I can sense the enthusiasm radiating from Orlando!
>
> I am just wondering if PDI really needs this ability to install modules dynamically without a restart.

[codeoncoffee]

The ability to install without a restart is just a side-effect of the
environment if properly setup. There is extra effort required to track
these comings and goings and I agree that it's not necessary, despite
the coolness factor

-Nick Baker

On Nov 30, 1:22

[dmoran]

Tony, I agree it might not be an absolute must have for spoon but it
does add to the overall user experience. Especially if you are in the
middle of developing a transform and realize you need a new step. It
sure would be nice to be able to grab it and keep going. I'd like to
see it as little more than another pallet to choose from. Another
interesting case would be if you tried to open a transform that you
didn't have the step or steps for, it would be sweet to have a prompt
that says, "This transform requires the CoolAndNewStep, do you want to
install it now? [yes] [no] [more info...]"

We should strive for seamless, especially if it comes with the
framework.

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Matt Casters]

Kettle is traditionally open to change and open to contributions. This fact
has not gone unnoticed even by our own developers at Pentaho.
I like the fact that Kettle is being used to experiment with OSGi since it's
a testament to a simple and straightforward architecture and I'm sure we'll
all welcome the contribution the once it's done. Nick's aim is to make this
not a replacement to the current plugin architecture (that Nick & I re-wrote
in v4) but to be complimentary.

However, Tony's point remains valid. OSGi is currently a technical solution
looking for a problem that Kettle might not even have. If we would
currently have a requirement to install a plugin without the need to restart
Spoon, I'm sure it would fairly easy to do. The Kettle plugins are already
dynamically loaded after all.

In other words, let's retain your user experience requirements and add them
to the rest and at the end we can see where we end up. Only then should we
determine whether or not OSGi is needed for Kettle and/or for the rest of
the Pentaho stack.

In addition to the technical capabilities and the things we want as
developers, there might be features that are not always desirable. One of
the "features" I seriously dislike about certain frameworks (Eclipse, Talend
and all) is the ability to automatically update. While technically a great
idea, it presents some challenges as far as conformance and testing are
concerned. Likewise, whatever plugin or market system we use, versioning is
something we need to do very carefully. If we support automatic updates,
allow for companies to turn it off or manage it centrally. Perhaps we want
to support development, test, UAT and production-like environments for
certain plugins? For other plugins a simple major, minor, point-version
system would work fine. The plugins that are downloaded in Spoon, do we
automatically want to deploy them in other locations too, signal Carte/DI
Servers to download the same package? What about firewalls? Can we tell
Spoon to download plugins from a shared folder or local server?

There are many important questions that need to be asked and answered at
this stage.

If we don't ask these questions, and we would adopt Nicks work at this time,
we would be putting a lot pressure on Nick to bend to the will of a
constantly changing set of requirements and that hardly seems fair. All I'm
saying is that the better the requirements we can gather are, before choices
have been made, the easier it will be to phase the development and execute
upon it later.

Finally, it might make sense to have the exact same discussion in the rest
of the Pentaho projects. I'm sure this would be a killer feature for the BI
server & report designer as well.

Thanks for caring!

Matt


2010/11/30 dmoran <dmoran (AT) pentaho (DOT) org>

>
> Tony, I agree it might not be an absolute must have for spoon but it
> does add to the overall user experience. Especially if you are in the
> middle of developing a transform and realize you need a new step. It
> sure would be nice to be able to grab it and keep going. I'd like to
> see it as little more than another pallet to choose from. Another
> interesting case would be if you tried to open a transform that you
> didn't have the step or steps for, it would be sweet to have a prompt
> that says, "This transform requires the CoolAndNewStep, do you want to
> install it now? [yes] [no] [more info...]"
>
> We should strive for seamless, especially if it comes with the
> framework.
>
> --
> You received this message because you are subscribed to the Google Groups
> "kettle-developers" group.
> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> To unsubscribe from this group, send email to
> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> For more options, visit this group at
> http://groups.google.com/group/kettle-developers?hl=en.
>
>


--
Matt Casters <mcasters (AT) pentaho (DOT) org>
Chief Data Integration, Kettle founder, Author of Pentaho Kettle
Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
(Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
Pentaho : The Commercial Open Source Alternative for Business Intelligence

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[dmoran]

I agree with Matt 108%

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[codeoncoffee]

I agree with you Matt. OSGI provides a wealth of options that aren't
all desirable at the same time. Sitting down and deciding how to
handle installation and configuration of modules across all of the
platform is key to determining the direction to take.

Last night I got the MaxMind step installed in Spoon by connecting to
a Bundle Repository running on my box. It automatically installed the
bundle I told it to and it's needed dependencies. Though this happened
over the web it could have installed the files from a shared place on
the filesystem or even a Maven repo. Pretty neat, plus I didn't write
any code.

How to handle system configuration for deployment is absolutely key.
One of the great features of Apache Aries and Apache Karaf (just
bundles you can add) is they allow you to work with "assemblies" of
bundles. These can be locked configurations of the system installed in
bulk. This may be a good approach to production deployment. There's
just too many options to list and I'm discovering more as I continue
this work.

One thing to keep in mind with the OSGI environment is that our users
can customize the entire deployment strategy by using existing OSGI
bundles available in the community. The Karaf extensions allow you to
manage SSHing directly into your running application and stop/start/
install/etc. the running Bundles. I certainly wouldn't ship a system
like this but could imagine wanting to do this as a user.

I've been running Spoon with the Apache Felix Web Console which embeds
a Jetty server running in your application with a web-app front-end to
manage the system. Certainly a lot of overhead for configuring Spoon,
but again, many options.

I would love to schedule some time to discuss the needs of our
individual products and the platform as a whole to try to find a
unified approach. We were hoping to get further along and learn a lot
more before presenting this initiative. I just couldn't wait anymore
when I heard of this Marketplace idea!

My immediate work will be on a walkthrough for creating a OSGI-
provided steps focused on step developers, but if anyone has any
specific questions or would like some pointers to examples, let me
know.

-Nick B

On Nov 30, 2:25

[Nicholas Goodman]

On Nov 30, 2010, at 12:19 PM, codeoncoffee wrote:

> I agree with you Matt. OSGI provides a wealth of options that aren't
> all desirable at the same time. Sitting down and deciding how to

With OSGi you get many things "for free" just by using it. The other thing one gets is lots of other smart people sorting out complicated issues like applications with depedencies across class loaders, etc.

For instance, in a service engagement we're building a set of PDI Job plugins. The plugins all have some common functionality (base class) and then we extend them for the specific fcnality. Given the current plugin classloading mechanism, we can't actually install all this as a plugin.

You can only have 1 plugin per plugin.xml, and each plugin has it's own classloader. Currently, you can't have a package (with a base class) either duplicated across all plugins (you'll get strange errors like Object X cannot be cast to Object X) or only in one (class not found from one plugin to another). The only way currently to solve this problem is to include the base classes, which are really part of the plugins, in the kettle libext and then have the individual plugins extended from that classloader.

So... there's one piece of functionality that OSGi solves well. The ability for plugins to build "on top of each other" instead of just being able to build upon core Kettle. Enough to warrant a whole framework? Maybe not. All the same, use cases for the way OSGi handles these types of things are there.

Nick

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Matt Casters]

Hi Nick,

Your assertions are fortunately not true.
Have a look at the KFF source code (http://is.gd/iNJ0b) for an example on
how you can easily deploy a series of plugins.
In that particular case we deploy 8 plugins (job entries and steps) from a
single jar file.
Shared dependency jar files can be placed in an adjacent lib/ folder.
And best of all: no XML is involved ;-)

Cheers,

Matt

2010/12/15 Nicholas Goodman <ngoodman (AT) bayontechnologies (DOT) com>

> On Nov 30, 2010, at 12:19 PM, codeoncoffee wrote:
>
> > I agree with you Matt. OSGI provides a wealth of options that aren't
> > all desirable at the same time. Sitting down and deciding how to
>
> With OSGi you get many things "for free" just by using it. The other thing
> one gets is lots of other smart people sorting out complicated issues like
> applications with depedencies across class loaders, etc.
>
> For instance, in a service engagement we're building a set of PDI Job
> plugins. The plugins all have some common functionality (base class) and
> then we extend them for the specific fcnality. Given the current plugin
> classloading mechanism, we can't actually install all this as a plugin.
>
> You can only have 1 plugin per plugin.xml, and each plugin has it's own
> classloader. Currently, you can't have a package (with a base class) either
> duplicated across all plugins (you'll get strange errors like Object X
> cannot be cast to Object X) or only in one (class not found from one plugin
> to another). The only way currently to solve this problem is to include the
> base classes, which are really part of the plugins, in the kettle libext and
> then have the individual plugins extended from that classloader.
>
> So... there's one piece of functionality that OSGi solves well. The
> ability for plugins to build "on top of each other" instead of just being
> able to build upon core Kettle. Enough to warrant a whole framework?
> Maybe not. All the same, use cases for the way OSGi handles these types of
> things are there.
>
> Nick
>
> --
> You received this message because you are subscribed to the Google Groups
> "kettle-developers" group.
> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> To unsubscribe from this group, send email to
> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> For more options, visit this group at
> http://groups.google.com/group/kettle-developers?hl=en.
>
>


--
Matt Casters <mcasters (AT) pentaho (DOT) org>
Chief Data Integration, Kettle founder, Author of Pentaho Kettle
Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
(Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
Pentaho : The Commercial Open Source Alternative for Business Intelligence

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Nicholas Goodman]

On Dec 15, 2010, at 11:05 AM, Matt Casters wrote:

> Your assertions are fortunately not true.
Love it when I'm wrong and it means a limitation isn't actually!!!

> And best of all: no XML is involved ;-)
Yeah... I saw some notes in the plugin loader that this was only for testing... Double checking looks like the general classpath search was the one that's marked as testing only. I got the sense that the annotations was the slightly off color method and that the plugin.xml was the primary way to build plugins. Are annotations now the preferred method?

Does the annotation method bring in all annotated plugins into separate classloaders? ie, with XML you get one classloader per plugin. Now, it would seem you get either
1) a single classloader for all annotated plugins
-or-
2) a single classloader per loaded .jar in plugins/ dir

This is great news! Makes my day! Multiple plugins in a single jar!

Nick

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Matt Casters]

Nick, there is no "preferred" method as we decided to support both methods
of plugin development.
However, since v4 it's obviously a lot more convenient to use annotations
for its ease of use.
There was a slight drawback to developing in Eclipse as documented over
here: http://is.gd/iNMn1
However, even that beats restarting Spoon a few hundred times to get a
plugin developed ;-)
The ease of use of just being able to unzip a file in plugins/ or drop a jar
file in there was worth the effort we put into it I think.

Plugins are added to the collective by whichever way works for you:
plugin.xml, annotations, kettle-steps.xml, kettle-jobentries.xml and one day
OSGi. From there on they play their role the same like all steps and job
entries.
And yes, they do indeed all use separate class loaders (Except for the SAP
Input step I believe)

Cheers,
Matt

2010/12/15 Nicholas Goodman <ngoodman (AT) bayontechnologies (DOT) com>

>
> On Dec 15, 2010, at 11:05 AM, Matt Casters wrote:
>
> > Your assertions are fortunately not true.
> Love it when I'm wrong and it means a limitation isn't actually!!!
>
> > And best of all: no XML is involved ;-)
> Yeah... I saw some notes in the plugin loader that this was only for
> testing... Double checking looks like the general classpath search was the
> one that's marked as testing only. I got the sense that the annotations was
> the slightly off color method and that the plugin.xml was the primary way to
> build plugins. Are annotations now the preferred method?
>
> Does the annotation method bring in all annotated plugins into separate
> classloaders? ie, with XML you get one classloader per plugin. Now, it
> would seem you get either
> 1) a single classloader for all annotated plugins
> -or-
> 2) a single classloader per loaded .jar in plugins/ dir
>
> This is great news! Makes my day! Multiple plugins in a single jar!
>
> Nick
>
> --
> You received this message because you are subscribed to the Google Groups
> "kettle-developers" group.
> To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
> To unsubscribe from this group, send email to
> kettle-developers+unsubscribe (AT) g...oups (DOT) com>
> .
> For more options, visit this group at
> http://groups.google.com/group/kettle-developers?hl=en.
>
>


--
Matt Casters <mcasters (AT) pentaho (DOT) org>
Chief Data Integration, Kettle founder, Author of Pentaho Kettle
Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/dp/0470635177>
(Wiley <http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>)
Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
Pentaho : The Commercial Open Source Alternative for Business Intelligence

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com.
To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) g...oups (DOT) com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[codeoncoffee]

Quick update. I've just committed a revised approach to getting OSGI
in Kettle. This new implementation is completely hidden behind the
current PluginRegistry api. With this commit, OSGI is just another
avenue to getting your plugin registered with the current system.

I will be adding a listener mechanism to the PluginRegistry so Kettle
can become aware as plugins come and go and react appropriately. This
provides the hot-swapping and loading.

The rest of the functionality added by OSGI will be available to the
plugin developers through traditional OSGI APIs. The MaxMind example
step uses the Blueprint beans approach right now. So far I'm only
registering Spoon as a service object but others will surely come.

I'm reasonably satisfied with this current effort. I'd like to see
more types proven-out other than steps, scheduler interface for
instance would be a nice demo. And the current directory watching
system needs to be replaced with Apache Karaf or just removed.

-Nick B.

On Dec 15, 2:28

[codeoncoffee]

The wiki has been updated with samples from the MaxMind Plugin:
http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle



On Dec 21, 8:55

[codeoncoffee]

The wiki has been updated with samples from the MaxMind Plugin:
http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle

On Dec 21, 8:55