Hitachi Vantara Pentaho Community Forums
Results 1 to 9 of 9

Thread: Advice On Rules

  1. #1

    Default Advice On Rules

    Hi all,

    We are trying to solve our legacy rules issues. We have multiple clients who require rules processing of their data to assign items to different people in their company. It's pretty simple and usually just based on postal code. The solution we have now is incredibly ugly brute force Javascript if/then implementation. But it does work.

    We have been looking into different Rules engines but I did do a proof of concept just pasting the rules into the Modified Javascript Step, setting the optimization level to -1 (some of our clients will produce 15k lines of if/thens) and it worked as well. What I'm wondering is

    1. Can you run a javascript include in the Modified JS Step? We would have to regenerate the rules.js file (for example) every time someone changed the rules from our front end to incorporate the rules in our transformations as is but is it possible to import/include a js file? Is it just wrong to try and run 1000s and 1000s of lines of code from the JS step? I did a POC on about 3000 rules and it seemed to work okay. We know the solution is ugly but if we can get it to work without rebuilding the entire engine that would be very good logistically and business wise.

    2. Is there a much better way to solve this problem within kettle (or outside it really, but I like kettle, so...)? We are moving to a .NET platform otherwise I think I would just start learning how to use Drools and be done with it. I know there's a Drools step but I was told here that it's not ready for production.

    We've been floundering a bit trying to solve this problem. Any advice would be greatly appreciated.

    Cheers,

    Walter.

  2. #2

    Default

    Anyone have a solution for this?

  3. #3
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    rules processing of their data to assign items to different people in their company. It's pretty simple and usually just based on postal code
    Maybe I'm oversimplifying this, but can't you use a combination of filter steps or switch/case steps?
    You can probably use lists to use in combination with variables to make this process semi-automatic if some people changes.
    -- Mick --

  4. #4
    Join Date
    Nov 2008
    Posts
    777

    Default

    Have you thought about:
    1. Setting up a web service (perhaps with OpenRules or JBoss Drools) and calling that service from a transformation?
    2. Embedding a rules engine in a User Defined Java Class?
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  5. #5
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Quote Originally Posted by codex View Post
    Can you run a javascript include in the Modified JS Step?
    If you want to keep it simple, try LoadScriptFile() in MJSV.
    So long, and thanks for all the fish.

  6. #6

    Default

    Quote Originally Posted by Mick_data View Post
    Maybe I'm oversimplifying this, but can't you use a combination of filter steps or switch/case steps?
    You can probably use lists to use in combination with variables to make this process semi-automatic if some people changes.
    Hmm... perhaps? Our clients have 20-30 thousand rules we have to check for at a time though. It may work, but may not? Would you mind elaborating? Is there a generic way to do that?

  7. #7
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    Hi Codex.
    If your client has 20k rules then probably the easiest approach would be the one suggested by other forum members.
    Mine approach should be chosen if you want to "translate" all those rules into "Pentaho - PDI" language.
    Obviously it would take you much longer to do it.
    Without looking at some of those "rules" I cannot really suggest anything more specific, I'm afraid.
    Your approach in solving this, it also depends on how you receive the data.
    For example, do you need to check each row of data for any of those 20K rules?
    Or do you receive a file and all rows have to be checked against the same group of rules (let's say only 200)?
    -- Mick --

  8. #8

    Default

    Walter,

    I have written a ruleengine in Java and a step for PDI, which allows you to check data or even update data based on rules. rules can be combined to complex scenarios. there is also a web application available to write the rules and actions, so a user (superuser) could do that instead of IT. the rules can be combined into projects containing many rules which will run to check the data, to filter the data or even for interfaces to other systems.

    you will not need to code anything - just write the rules. but the system is open so that you could extend what I call "checks" and also the actions, if that would be a requirement.

    the step is in the marketplace and on github. the webapplication is also on github, but I have no documentation yet. I will do that the next days.

    rgds,

    uwe

  9. #9

    Default

    one more thing maybe:

    I have also implemented a socket server, running the rule engine. you could run it on a server. I have also done some tests running it in docker containers.

    you send the data to the server and it will run the rules and send back the results. but this is not inside PDI - I don't see (yet) how that would integrate into the workflow of an etl.

    hopefully I will present the ruleengine step and the web application for maintaining the rules at one of the community meetings.

    uwe

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.