Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Building PDI into a custom app.

  1. #1
    Join Date
    Jun 2007
    Posts
    233

    Question Building PDI into a custom app.

    Hi Everyone,

    I am not sure where exactly to post this question, but I thought here might be a great place to start. What I am trying to discover is how to start a project / work with a business group to develop a 'variant' of PDI that is customised around our workplace requirements.

    This cou´ld be done with another tool, then launching PDI via scripts, however this is not an idal solution for us. What we would, as a business, like to do is to have a tool that uses the PDI functionality to design and execute transformations / jobs and have these fitted in with a particular user interactive workflow process.

    Without going into the full specifications of the application design, does anyone have any suggestions on how this might be approached so that it can be supported externally to the company I am working for. We are looking at developing an ongoing business relationship to handle a controlled product development, implementation and lifecycle. Any ideas would be appreciated.

    We are a global company, but the particular office I am working from is located in Germany, so this would have to be able to accomodate multiple languages, starting with English and German.

    Cheers

    The Frog
    Everything should be made as simple as possible, but not simpler - Albert Einstein

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    It all depends... . You have the PDI source code, it is LGPL.... just do it

    It depends on what you want to integrate it with, and to which extent. Several people have integrated the PDI engine into their own applications looking at some of the questions on this forum. One of the lacking things at the moment e.g. is 100% good general logging functionality (there's a still a JIRA out for it).

    If you just want to use Spoon to make jobs/transformations, and integrate the execution itself it will probably be "easy": strip pan and kitchen and start from there e.g. If you want to go further and integrate spoon itself with your own GUI stuff you're in for a harder integration path.
    Even if you go the simple path, PDI is a "moving target"... and it may take additional time to keep up with PDI. E.g. some people have integrated PDI 2.5 into their applications and they have to redo some things when they want to go to PDI 3.x (interface changes, ...). It's highly likely you have to redo things between PDI versions to keep it working.

    For fun I did just a repackaging of PDI and that already takes time to maintain.

    Regards,
    The Parrot

    P.S. I don't think there's life on Mars
    Last edited by sboden; 08-18-2008 at 06:39 AM.

  3. #3
    Join Date
    Jun 2007
    Posts
    233

    Exclamation Integration of PDI

    Hi there Parrot(!)

    Thanks for getting back to me so quickly. The situation we are looking ta is one of maintainability. While at the moment I am chiefly responsible for development in my department, and I help with the development process for other departments too, I will not be here forever. Rather than risk the knowledge loss and re-learning time we would be better of having a well developed specification for a 'product' that meets our needs, and simply employ a suitable group to do the development work and keep them ona retainer / contract to keep things up to date and help troubleshoot.

    In short the business doesnt want to go down the path of getting into development 'wholsale', and would rather outsource the development process but keep a hand on the controls. It would take me a lot less to develop a 'living' specification for a product than it would to train and educate my 'successor(s)' in the whys and ways of BI, Java, ETL, etc..... when they are most likely only going to have a limited perspective on such things. In short we want to deal with the meta-data of the business process and development rather than the development itself.

    I am nearing the end of a development cycle now, and am hoping to produce a prototype 'core' of information / data handling that the department can use as the basis for its work. For this to work effectively int he future it will require three things:
    1/ A clear definition of the 'data core' and 'BI' (including ETL) capabilities and how they work
    2/ A clear definition of the workflow processes (department specific) that need to use these capabilities provided by step 1
    3/ Budget (ie/ an ongoing business case) to keep the development rolling and the support ongoing.

    Definition of step 1 is the development step I am nearly finished with (the prototype), and I am hoping that PDI can continue to be a significant part of that, but it is needing some 'work' specific things that at the moment I manage by hand.

    Step 2 is already a well defined process and set of practices that were in fact used to define what was needed for step 1. There are some market specific tools that could sit nicely on top of the Pentaho BI Suite or even just Mondrian that would give us a nice performance boost - but we cross that bridge after step 1

    Part 3 is already a well developed and accepted business case. The performance and out uplift, not to mention the impact on turnover is significant enough that we received senior managerial approval in less than 20 minutes for a project. Its a solid case with a high potential return with minimal risk.

    As much as I would love to do this myself, and take the time to do it, this simply cannot be in the business context. What we need is a group to undertake the technical / development side, and let the staff get on with the work they are employed to do. Once I have 'delivered' the prototype, then in theory my job is done and we need to move to a maintainable deployment scenario.

    I am hoping that with the experts and business people and developers that peruse here that there may be a way to continue beyond the prototype into a full deployment using the Pentaho technology.

    Any thoughts? Is there a B2B forum for sourcing such things?

    Cheers

    The Frog
    Everything should be made as simple as possible, but not simpler - Albert Einstein

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    I'm sure the pentaho sales people can give you a quote for customization work ... for the rest there are some Pentaho partners spread out over the world, but I don't know whether and how they offer customizations.

    If you do it yourself or you have it done if it's not in the trunk of PDI you will have work later on with a newer PDI version. And depending how deep you want to integrate having something in the PDI trunk will be difficult.

    Consider it similar to a SAP implementation ... "easy" if you stick with the defaults, nightmarish if you want to custom fit SAP to your business processes.

    Regards,
    Sven

  5. #5
    Join Date
    Jun 2007
    Posts
    233

    Default PDI Integration into app

    Hi Sven,

    Thanks again for getting back to me so quickly. I think it is time that we (as a company) spoke to the sales team to see what is involved. I will take this up with my boss and see what he says, but I think it should be fine. The 'customisation' I am asking for would not be particularly difficult I think, simply using pre-defined transformations / jobs designed in PDI, 'loaded' into a machine with the sole purpose of doing the ETL, and then activating the various ETL processes as required by the users when they work through their workflow. Its really used as a basis for ensuring data quality, so that each new item of data that is integrated into the dimensions actually has to be approved by the team that is working with it. This is especially important to us with regards to product definitions as ours are very purpose specific.

    Incoming data approved, etl process is activated and the approved data is integrated. Backup needs to be done before integrating new data too so that a form of 'rollback' is possible. I think this should be pretty straight forward for experienced developers and a proper product spec to build to. Thanks for your help Sven, I appreciate it.

    Cheers

    The Frog

    PS: Life does exist on Mars, its just not normal
    Everything should be made as simple as possible, but not simpler - Albert Einstein

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.