US and Worldwide: +1 (866) 660-7555
Results 1 to 8 of 8

Thread: control version - how can i implement version control to KETTLE packages and jobs

  1. #1
    Join Date
    Dec 2006
    Posts
    27

    Default control version - how can i implement version control to KETTLE packages and jobs

    hi folks,

    i have been asked if kettle (pdi) has a version control system and to be honest, i don't know.

    is there a way of having version control on kettle?


    thanks a lot,


    Nicolas

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    It's not in v3.1, it does seem to be on someones "todo"-list of a future version.

    So for the moment all you can do is version the xml files outside of Spoon.

    Regards,
    Sven

  3. #3
    Join Date
    Nov 1999
    Posts
    9,688

    Default

    I would have started to work on this as it's a priority for us. However, version support is something that transcends the Pentaho stack.
    As such, we will have a chat about it in 2 weeks on our development summit in Orlando.

    It would be cool if we could abstract it and allow usage of all sorts of version control and CMS systems (Alfresco et all).
    Anyway, it should be fun to do this.
    Matt Casters, Chief Data Integration
    Pentaho, Open Source Business Intelligence
    http://www.pentaho.org -- mcasters@pentaho.org

    Author of the book Pentaho Kettle Solutions by Wiley. Also available as e-Book and on the Kindle reading applications (iPhone, iPad, Android, Kindle devices, ...)

    Join us on IRC server Freenode.net, channel ##pentaho

  4. #4
    Join Date
    Dec 2006
    Posts
    27

    Default oops

    thanks Matt,

    i am suggesting to a company that i work with to adopt kettle (pdi).
    but one of their top requirements is CVS.

    would it be realistic to ask for some idea of time scale when this feature might appear?

    amongst the options for version control; will there be one that keeps track of version in the repository that is currently used by kettle?

    thanks,

    Nicolas

  5. #5
    Join Date
    Nov 1999
    Posts
    9,688

    Default

    We already support it since you can simply check in your XML into CVS.
    Kettle doesn't have any "Packages" by the way.

    The upside of that solution is that this company can use whatever it is using today and doesn't need anything new as far as software is concerned.
    The downside is simply that we haven't integrated versioning into Spoon yet.
    Matt Casters, Chief Data Integration
    Pentaho, Open Source Business Intelligence
    http://www.pentaho.org -- mcasters@pentaho.org

    Author of the book Pentaho Kettle Solutions by Wiley. Also available as e-Book and on the Kindle reading applications (iPhone, iPad, Android, Kindle devices, ...)

    Join us on IRC server Freenode.net, channel ##pentaho

  6. #6
    Join Date
    Oct 2007
    Posts
    255

    Default asdf

    I'll give you my opinion on this, but don't take it as 'the best solution' -- it just works for me, maybe for you.

    I found Kettle's existing repository very useful for some things (eg, consistent list of databases, able to create a consistent folder structure for storing/referencing jobs & transformations), I had [mostly minor] difficulties in other regards. The main thing that convinced me to use subversion was, as Matt said, using an existing repository & versioning system.

    I liked the ability to ignore file system structure in the Kettle repository, so the first hurdle to overcome was that, through variables in Kettle (~/.kettle/kettle.properties, or a similar path for windows).

    The database list I don't have a 'great' solution for -- the Kettle respository is superior in this regard, at least in my experience. You can use shared databases in Kettle without the repository, but all transformations/jobs get populated with available connections. If you move the scripts to a machine without copying your shared.xml file over, I've encountered consistency issues with some embedded database connections across jobs/transformations, which is only a problem until you populate a local shared.xml file and get the settings correct. I've been meaning to see if it's possible to specify a shared.xml through a parameter or something similar, so that could be tracked in a versioning system in a relatively easy fashion.

    As I said, this has just worked for me, I don't know if it's necessarily the best approach.

    -Brian

  7. #7
    Join Date
    Mar 2008
    Posts
    101

    Default

    Quote Originally Posted by MattCasters View Post
    I would have started to work on this as it's a priority for us. However, version support is something that transcends the Pentaho stack.
    As such, we will have a chat about it in 2 weeks on our development summit in Orlando.

    It would be cool if we could abstract it and allow usage of all sorts of version control and CMS systems (Alfresco et all).
    Anyway, it should be fun to do this.
    Matt, is the development summit a Pentaho employee only event or is this a public event. Just wondering. If not, does Pentaho have any annual public events?

    Here is my 2-cents regarding version control as I just went through this for our implementation:

    I first began by setting up a development kettle repository and then a separate test kettle repository. My initial idea was that I could control the code promotion to the test kettle repository using the kettle user permissions (i.e. read/write in dev, only read in test) using the kettle repository. However, as you know there is no integrated version control support. So, I bagged that idea and no longer use the kettle repository.
    The source control product that we use is git. I set up a git repository for the kettle jobs/transformations so that I can check them in/out etcetera. This seams to work quite well. I like the fact that I can use any version control system and hope that would remain as you determine what direction to head.

  8. #8
    Join Date
    Nov 1999
    Posts
    9,688

    Default

    Our Pentaho Developer summit is an architects summit to decide on things that matter to Pentaho.
    To be honest, the goals of Pentaho and the community are usually pretty much in line.

    As far as public events, we had a first Community Event in Germany earlier this year. Since that was a big success there are going to be others for sure.

    I like the fact that I can use any version control system and hope that would remain as you determine what direction to head.
    Don't worry, I don't see how we could rip out XML support out of Kettle, even if we wanted to :-)

    Take care,
    Matt
    Matt Casters, Chief Data Integration
    Pentaho, Open Source Business Intelligence
    http://www.pentaho.org -- mcasters@pentaho.org

    Author of the book Pentaho Kettle Solutions by Wiley. Also available as e-Book and on the Kindle reading applications (iPhone, iPad, Android, Kindle devices, ...)

    Join us on IRC server Freenode.net, channel ##pentaho

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •