Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Re: Steps Categorization

  1. #1
    Sven.Boden Guest

    Default Re: Steps Categorization

    For the categories... there's already a kettle-steps.xml file in which you can reorganize the steps in different categories (before PDI distribution anyway).

    For the new categories... not sure... it does start to look cluttered but making more categories won't really fix that (not for long anyway). Maybe we should be thinking of other things... e.g. splitting the steps in "packs", separately installable.

    - Basic pack: table input, javascript, select values generate rows, text file steps, table output, update, insert/update, sort, unique, blocking, ...
    - Data warehouse pack: dimension, ...
    - Transformation pack 1: ...
    - Transformation pack 2: ...
    - Bulk loader pack: ...
    - Validation pack

    Regards,
    Sven

    >----- Oorspronkelijk bericht -----
    >Van

    : jianjun chu [mailto:jianjunchu (AT) gmail (DOT) com]
    >Verzonden

    : zaterdag
    , december
    6, 2008 08:08 AM
    >Aan

    : kettle-developers (AT) googlegroups (DOT) com
    >Onderwerp

    : Re: Steps Categorization
    >
    >Step categories is hard coded now, A config file may be need to resolve this
    >problem.
    >
    >such as:
    >
    ><categories>
    > <category name = "input">
    > <step name = "table input">
    > ...
    > </category>
    >
    > <category name = "output">
    > ...
    > </category>
    ></categories>
    >
    >
    >2008/12/6 samatar hassan <sahass78 (AT) yahoo (DOT) fr>
    >
    >> Hi all,
    >>
    >> I came along the steps categories in Spoon and i a propose to add 2 new
    >> categories :
    >>
    >> - Flow control :
    >> it's about all the steps that deal with flow and not fields. We have here
    >> the filter step for example.
    >> I will select :
    >> --> Filter step
    >> --> Sort
    >> --> Blocking step
    >> --> Append
    >> --> Row clone
    >> --> Switch
    >> --> Detect empty stream
    >> --> Sample rows
    >> --> Detect last row (Identify last row i n stream)
    >>
    >>
    >>
    >> - Data Quality :
    >> The best should come ..
    >> In mean time, i propose to put the following steps:
    >> --> Unique rows
    >> --> Add a checksum
    >> --> Mail validator
    >> --> Credit card validator
    >>
    >>
    >> At the end, nothing really new...but i am waiting for your remaks, idea,...
    >>
    >> Thanks and take care
    >>
    >>
    >> Samatar
    >>
    >>
    >>
    >> >

    >>

    >
    >>

    >




    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) googlegroups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

  2. #2
    samatar hassan Guest

    Default Re : Steps Categorization

    Hi,
    jianjun, Sven
    thanks for your feedbacks :-)
    Maybe organizing by packs will bring confusion on users mind..because they excepected to get all necessary steps in basic pack.


    Samatar




    ________________________________
    De : Sven.Boden <list123 (AT) pandora (DOT) be>
    À : kettle-developers (AT) googlegroups (DOT) com
    Envoyé le : Samedi, 6 Décembre 2008, 23h00mn 50s
    Objet : Re: Steps Categorization



    For the categories... there's already a kettle-steps.xml file in which you can reorganize the steps in different categories (before PDI distribution anyway).

    For the new categories... not sure... it does start to look cluttered but making more categories won't really fix that (not for long anyway). Maybe we should be thinking of other things... e.g. splitting the steps in "packs", separately installable.

    - Basic pack: table input, javascript, select values generate rows, text file steps, table output, update, insert/update, sort, unique, blocking, ...
    - Data warehouse pack: dimension, ...
    - Transformation pack 1: ...
    - Transformation pack 2: ...
    - Bulk loader pack: ...
    - Validation pack

    Regards,
    Sven

    >----- Oorspronkelijk bericht -----
    >Van

    : jianjun chu [mailto:jianjunchu (AT) gmail (DOT) com]
    >Verzonden

    : zaterdag
    , december
    6, 2008 08:08 AM
    >Aan

    : kettle-developers (AT) googlegroups (DOT) com
    >Onderwerp

    : Re: Steps Categorization
    >
    >Step categories is hard coded now, A config file may be need to resolve this
    >problem.
    >
    >such as:
    >
    ><categories>
    > <category name = "input">
    > <step name = "table input">
    > ...
    > </category>
    >
    > <category name = "output">
    > ...
    > </category>
    ></categories>
    >
    >
    >2008/12/6 samatar hassan <sahass78 (AT) yahoo (DOT) fr>
    >
    >> Hi all,
    >>
    >> I came along the steps categories in Spoon and i a propose to add 2 new
    >> categories :
    >>
    >> - Flow control :
    >> it's about all the steps that deal with flow and not fields. We have here
    >> the filter step for example.
    >> I will select :
    >> --> Filter step
    >> --> Sort
    >> --> Blocking step
    >> --> Append
    >> --> Row clone
    >> --> Switch
    >> --> Detect empty stream
    >> --> Sample rows
    >> --> Detect last row (Identify last row i n stream)
    >>
    >>
    >>
    >> - Data Quality :
    >> The best should come ..
    >> In mean time, i propose to put the following steps:
    >> --> Unique rows
    >> --> Add a checksum
    >> --> Mail validator
    >> --> Credit card validator
    >>
    >>
    >> At the end, nothing really new...but i am waiting for your remaks, idea,...
    >>
    >> Thanks and take care
    >>
    >>
    >> Samatar
    >>
    >>
    >>
    >> >

    >>

    >
    >>

    >







    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) googlegroups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

  3. #3
    bigjonroberts@gmail.com Guest

    Default Re: Re : Steps Categorization

    If I'm understanding correctly, the core issue is not really so much
    "how the steps are organized", but the "increasing number of steps."

    I find myself more and more searching through the list to find the
    step that I need.

    I think that there are a few different strategies to deal with this,
    and that they can all be applied in tandem.

    1. Tweak the categories and organization, including sub-categories?
    -- too much of this too often and users will get frustrated with you
    moving the furniture around all the time. But a needed refresh every
    so often could possibly be helpful.

    2. Recently used steps - already implemented in "History" category. I
    can usually quickly find many of my most commonly used steps there.

    3. A search tool - to find the step you are looking for. I've seen
    this same issue creep into RAD development tools that I've used with
    component pallets. One common solution applied to the problem is a
    text 'search bar" at the top of the component pallet. Usually giving
    visual feedback - matching components - listed as the text is typed
    in. Maybe this could be implemented in Spoon with a dynamic "Search"
    category that exposes the search text input underneath it when
    expanded. It could default to the top or bottom tree item. There
    might not quite be enough steps yet to really justify this, but if
    they keep multiplying like rabbits with every release, I suspect that
    time is not to far in the future. I'm not complaining, I like all the
    new steps and have been using some of them; but it does introduce this
    new problem.

    4. Using the (online?) documentation more effectively, or somehow
    integrating that better into the UI. This is really similar to the
    search tool, maybe a handy link to the steps documentation page that
    is always available in the left hand step pallet. Or that could be
    turned off and on as a config option.

    As a side note, I'd like to see the number of items appearing in
    "History" as a configurable option. I'll open a ticket for this if
    one doesn't already exist.

    On Dec 7, 4:51

  4. #4
    Matt Casters Guest

    Default Re: Steps Categorization

    Thanks for that feedback.

    I think it's actually pretty "cheap" to deliver more steps to our users, both from a development and maintenance perspective.
    Most of the time we see changes and bugs files against a select group of most used steps.
    A usual, 10% of the steps see 90% of the action.
    As such, I'm no longer sure that keeping certain steps out of the core distribution is the right solution.
    Sure there has to be a certain need for them, a reason for them to exists, but once we have a group of end-users, I'm fine with it.
    In the end, we do solve specific gnarly problems with a lot of these steps so there is added value in supplying them to our users.

    I was thinking of that step "Search" option as well. I'm not sure it's *the* solution all by itself, but it's a step in the right direction.

    So here is another idea: perhaps you can create a set of different cross sections with a drop down box like:
    - Database steps : selects all the database steps in the whole set of steps
    - String manipulation
    - Calculation
    - etc.
    Only those steps that match the criteria in the various categories would be shown.

    Other idea building on that: have a floating panel or toolbar somewhere that you can populate with your own set of favorite steps.
    Or even better: Have a set of these. If you work on the same type of transformations you want the same set to be shown.
    You can allow the user to populate that floating panel/toolbar but you can also pre-populate it. (see above)

    As far as the documentation is concerned, help buttons on the step dialogs are planned to go to the wiki, configurable like you mentioned.
    Time constraints have prevented me from doing the implementation though. There always seems to be something going on :-)

    Cheers,
    Matt




    On Monday 08 December 2008 10:10:25 bigjonroberts (AT) gmail (DOT) com wrote:[color=blue]
    >
    > If I'm understanding correctly, the core issue is not really so much
    > "how the steps are organized", but the "increasing number of steps."
    >
    > I find myself more and more searching through the list to find the
    > step that I need.
    >
    > I think that there are a few different strategies to deal with this,
    > and that they can all be applied in tandem.
    >
    > 1. Tweak the categories and organization, including sub-categories?
    > -- too much of this too often and users will get frustrated with you
    > moving the furniture around all the time. But a needed refresh every
    > so often could possibly be helpful.
    >
    > 2. Recently used steps - already implemented in "History" category. I
    > can usually quickly find many of my most commonly used steps there.
    >
    > 3. A search tool - to find the step you are looking for. I've seen
    > this same issue creep into RAD development tools that I've used with
    > component pallets. One common solution applied to the problem is a
    > text 'search bar" at the top of the component pallet. Usually giving
    > visual feedback - matching components - listed as the text is typed
    > in. Maybe this could be implemented in Spoon with a dynamic "Search"
    > category that exposes the search text input underneath it when
    > expanded. It could default to the top or bottom tree item. There
    > might not quite be enough steps yet to really justify this, but if
    > they keep multiplying like rabbits with every release, I suspect that
    > time is not to far in the future. I'm not complaining, I like all the
    > new steps and have been using some of them; but it does introduce this
    > new problem.
    >
    > 4. Using the (online?) documentation more effectively, or somehow
    > integrating that better into the UI. This is really similar to the
    > search tool, maybe a handy link to the steps documentation page that
    > is always available in the left hand step pallet. Or that could be
    > turned off and on as a config option.
    >
    > As a side note, I'd like to see the number of items appearing in
    > "History" as a configurable option. I'll open a ticket for this if
    > one doesn't already exist.
    >
    > On Dec 7, 4:51 am, samatar hassan <sahas... (AT) yahoo (DOT) fr> wrote:[color=green]
    > > Hi,
    > > jianjun, Sven
    > > thanks for your feedbacks :-)
    > > Maybe organizing by packs will bring confusion on users mind..because they excepected to get all necessary steps in basic pack.
    > >
    > > Samatar
    > >
    > > ________________________________
    > > De : Sven.Boden <list... (AT) pandora (DOT) be>
    > >

  5. #5
    Sven.Boden Guest

    Default Re: Steps Categorization

    I like the idea of configuring the steps you see... What about an option to choose step configuration... novice/intermediate/advanced (all?)/own configuration.

    Novice/inter../advanced are a hardcoded view of the steps. The moment you change something a configuration it becomes your own configuration.

    The step/job entry configuration could be saved in the .kettle directory and the GUI for it could look something like a dialog with on the left hand side the tree of steps/job entries... on the right hand side a list of all available steps. With some buttons to add steps, remove steps, ...

    Novice users would start out with a limited set of steps not to confuse them too much, long time users just can see all steps.

    Regards,
    Sven

    >----- Oorspronkelijk bericht -----
    >Van

    : Matt Casters [mailto:mattcasters (AT) gmail (DOT) com]
    >Verzonden

    : maandag
    , december
    8, 2008 11:26 PM
    >Aan

    : kettle-developers (AT) googlegroups (DOT) com
    >Onderwerp

    : Re: Steps Categorization
    >
    >
    >Thanks for that feedback.
    >
    >I think it's actually pretty "cheap" to deliver more steps to our users, both from a development and maintenance perspective.
    >Most of the time we see changes and bugs files against a select group of most used steps.
    >A usual, 10% of the steps see 90% of the action.
    >As such, I'm no longer sure that keeping certain steps out of the core distribution is the right solution.
    >Sure there has to be a certain need for them, a reason for them to exists, but once we have a group of end-users, I'm fine with it.
    >In the end, we do solve specific gnarly problems with a lot of these steps so there is added value in supplying them to our users.
    >
    >I was thinking of that step "Search" option as well. I'm not sure it's *the* solution all by itself, but it's a step in the right direction.
    >
    >So here is another idea: perhaps you can create a set of different cross sections with a drop down box like:
    >- Database steps : selects all the database steps in the whole set of steps
    >- String manipulation
    >- Calculation
    >- etc.
    >Only those steps that match the criteria in the various categories would be shown.
    >
    >Other idea building on that: have a floating panel or toolbar somewhere that you can populate with your own set of favorite steps.
    >Or even better: Have a set of these. If you work on the same type of transformations you want the same set to be shown.
    >You can allow the user to populate that floating panel/toolbar but you can also pre-populate it. (see above)
    >
    >As far as the documentation is concerned, help buttons on the step dialogs are planned to go to the wiki, configurable like you mentioned.
    >Time constraints have prevented me from doing the implementation though. There always seems to be something going on :-)
    >
    >Cheers,
    >Matt





    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) googlegroups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

  6. #6
    Roland Bouman Guest

    Default Re: Steps Categorization

    Hi All,

    Personally I think it would be best to always ship all publicly
    available steps, at least as long as there aren't hundreds of them.
    It's no fun to hunt for steps when you need them esp. when you are not
    in complete control of the machine where kettle is installed. I would
    prefer to be able to rely on everything to be there so that you'd only
    need to mind the program version.

    I like this idea of Sven to allow the UI to be configurable.

    Sorry for wandering off to what may be perceived as the most minor,
    least of worries issue:
    - why not use an XML format to declare the categorization (possibly
    supporting sub categories)
    - in the XML, use %KEY% placeholders for the labels, and property
    files for the localized values of keys (like what is used for the
    index.xml files used in the pentaho platform)

    just an idea.

    Kind regards,

    Roland

    On Tue, Dec 9, 2008 at 5:56 PM, Sven.Boden <list123 (AT) pandora (DOT) be> wrote:
    >
    >
    > I like the idea of configuring the steps you see... What about an option to choose step configuration... novice/intermediate/advanced (all?)/own configuration.
    >
    > Novice/inter../advanced are a hardcoded view of the steps. The moment you change something a configuration it becomes your own configuration.
    >
    > The step/job entry configuration could be saved in the .kettle directory and the GUI for it could look something like a dialog with on the left hand side the tree of steps/job entries... on the right hand side a list of all available steps. With some buttons to add steps, remove steps, ...
    >
    > Novice users would start out with a limited set of steps not to confuse them too much, long time users just can see all steps.
    >
    > Regards,
    > Sven
    >
    >>----- Oorspronkelijk bericht -----
    >>Van

    > : Matt Casters [mailto:mattcasters (AT) gmail (DOT) com]
    >>Verzonden

    > : maandag
    > , december
    > 8, 2008 11:26 PM
    >>Aan

    > : kettle-developers (AT) googlegroups (DOT) com
    >>Onderwerp

    > : Re: Steps Categorization
    >>
    >>
    >>Thanks for that feedback.
    >>
    >>I think it's actually pretty "cheap" to deliver more steps to our users, both from a development and maintenance perspective.
    >>Most of the time we see changes and bugs files against a select group of most used steps.
    >>A usual, 10% of the steps see 90% of the action.
    >>As such, I'm no longer sure that keeping certain steps out of the core distribution is the right solution.
    >>Sure there has to be a certain need for them, a reason for them to exists, but once we have a group of end-users, I'm fine with it.
    >>In the end, we do solve specific gnarly problems with a lot of these steps so there is added value in supplying them to our users.
    >>
    >>I was thinking of that step "Search" option as well. I'm not sure it's *the* solution all by itself, but it's a step in the right direction.
    >>
    >>So here is another idea: perhaps you can create a set of different cross sections with a drop down box like:
    >>- Database steps : selects all the database steps in the whole set of steps
    >>- String manipulation
    >>- Calculation
    >>- etc.
    >>Only those steps that match the criteria in the various categories would be shown.
    >>
    >>Other idea building on that: have a floating panel or toolbar somewhere that you can populate with your own set of favorite steps.
    >>Or even better: Have a set of these. If you work on the same type of transformations you want the same set to be shown.
    >>You can allow the user to populate that floating panel/toolbar but you can also pre-populate it. (see above)
    >>
    >>As far as the documentation is concerned, help buttons on the step dialogs are planned to go to the wiki, configurable like you mentioned.
    >>Time constraints have prevented me from doing the implementation though. There always seems to be something going on :-)
    >>
    >>Cheers,
    >>Matt

    >
    >
    >
    >
    > >

    >




    --
    Roland Bouman
    http://rpbouman.blogspot.com/

    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers+unsubscribe (AT) googlegroups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.